Optimization Horizons: system and algorithmic trading
New horizons of optimization
Work in progress
By jaguar1637 4824 days ago Comments (24)
Hi
Well, I am on the way on 3 # projects
- to rewrite Burg w/ RFSDS
- to write a swarm particule optimization indicator
- to write something w/ Ornstein-Uhlebeck algo
I will show you the rest, when my computer will be repaired
jy
Comments
I was thinking that you were working on goertzel browser too?
I was thinking how to make a better connection between rapid miner and metatrader. I am learning slowy how to use this wonderful piece of software.
yes, this is true.
The connection between rapidminer and metatrader is quite difficult. the best, is to fetch the 15 minutes Time series, put them inside a .csv file, analyse it w/ rapideminer or something like, get back the prediction and go back to metatrader and open a trade
During quite 5 minutes, the market can move very fast !
So, I try from my side to connect directly R-Code.
If this works, there will be no disconnection, I hope, during the prediction process
The other thing is to use the expert in metatrader. The expert writes the data into csv file. This files is located in experts/files folder. In rapidminer I use the Read CSV oparator and I open the file in the Metatrader folder experts/files. And it is loaded in Rapid miner. Then I just click the Run process button once per bar in Metatrader or even sooner as the information is authomatically updated.
In fact you did the mods for the transfer. However I am wondering how I could limit the bars that would be processed in the folder to a level, 1000 for example because I may not need the whole history.
Here is the indicator, in fact this is your mod.
Hi, In my humble opinion, to get exactly the right numbers of bars to be analyzed , is to get it from the Quantum indicator, Moving Min Max shifted, becoz it shows exactly the sequence to look for
So, this indicator should be rewrite to return this value. the problem If this value is less than 20 or 40 bars, something else should be found to fetch the right value
I believe instead of using Rapid miner much easier is to use matlab with MT4. It can communicate with MT4 in real time either though csv files or though DDE, my system is using for 1 min charts with no problem. Main advantage of Matlab is that a lot of software is ready, all universites are using it, all TA indicators are also implemented there.
I already tried a lot of platforms like Neuroshell, Tradestation/Multicharts and of course MT4 and I think this combination (MATLAB+MT4) is the most powerfull
Yes I know the power of Matlab. Nobody can argue that that is the industry standard.
However Rapid Miner is free. Rapid miner has no add - ons everything is there. Well that is not really expensive but why the average trader has to pay in order just to try to use something that can be free.
With the tutorials of Tom no coding experience is needed in order to design a complex system.
My idea is, if it is successful to make the bridge between Metatrader 4 and Rapid Miner. And then we can use the complex indicators that were created for Metatrader using the State of the Art learners.
I hope somebody can find a successful model and share as I did not for the moment.
And something more. In Rapid Miner there is a new add - on allowing for a classification model to appreciate what is the best model and authomatically construct a model and optimize the model with just some clicks: Pattern Recognition & Landmarking plugin
All this means that a potential community could be made arround the connection MT4 - Rapid.
1. The two tools are free to use for the end - user. (Without the freedom to use we cannot build a large community. The owner of metaquotes is a multi millionaire by now knowing this.)
2. They do not need coding skills (this is relative).
3. Mt 4 has a huge library of indicators. Mt 4 has just one defective genetic optimizer
4. Rapid miner has a lot of learners and data discovery tools.
See my post on SVM blog...if you are a very bad guy you can always download MATLAB from piratebay...
Yes I am, but no successful results so far with Neuroshell or Trading solutions.
Your last post Theo, let me in deep thoughts and perplex. As far as I understand, we did not find the right value or price calculation to analyse
I thought you fetch something attractive in SSA value or PFE.
also, we have only the Moving Min Max to be sure (and the correlation matrix dimension) to know the lenght of the time series to analyse
and regarding the algorythm to use (from GA, NN, BPNN,NeuroShell, Burg, PSO), you say that there is no tools able to claim to be the best
All the best
(I will continue to work on my side, this afternoon to make this project go ahead)
There is the no free lunch theorem. "The no free lunch theorem for search and optimization (Wolpert and Macready 1997) applies to finite spaces and algorithms that do not resample points. All algorithms that search for an extremum of a cost function perform exactly the same when averaged over all possible cost functions. So, for any search/optimization algorithm, any elevated performance over one class of problems is exactly paid for in performance over another class."
I am the point at which I think that the key is to simplify and simplify things:
1. Using an algorythm that extracts only the most important characteristics of the time series.
SSA, Hodrick Prescott are doing that.(We have a vast library of algorythms under metatrader)
2. Apply a learner on those series.
3. Time series matters, as I am not able to use high frequency data and have a really nice execution and speed (even if I had I have not the competence to use that properly). So I am trying to adjust the model on very specific market data that has some kind of homogeneity.
I use the model based on:
-pattern recognition based on volatlity and fractal dimension.(There is a page on TSD about that combination beween Kurt Faith market states and fractal dimension) and/or
-human pattern recognition based on Elliott wave principle (Elliotware approach).
So far really good predictions. Still testing. And I am still there.
For example look that .
This is THE ACTUAL MONEY MACHINE.
It is based on specific market conditions that exist because of a fundamental reason. Check the graph.
That is all we can do looking for opportnities. The oppotunities arise from time to time. Under the term opportunity I mean a completely deterministic market behaviour. When we have a complety deterministic market behaviour the choice of the algorythm may be very, very simple and is based on the personnal experience.
I do not believe we can beat anytime anywhere the market. Some market conditions give us better chances.
As my mentor told me that it is important to learn to take small amount of pips every day, however to be under a constant watch for market opportunities. And a trader's network is all that about sharing the opportunities.
I've been using the tool found in this thread to connect my programming language with the R project. This interface was developed to connect MT4 with R (open source / free to download).
http://www.forexfactory.com/showthread.php?t=260422
An interesting application of this interface is below:
http://www.forexfactory.com/showthread.php?t=262827
My take is that while in the 2nd thread the attempt was made to create a deterministic cointegrated trending channel, the method used to achieve this had some basic math flaws (trend-o-mat). However, the idea is not too far off the mark IMO. The arb-o-mat implementation looks sound enough. This is in the realm of stat arb and may or may not be of interest to members of this group / forum.
The guy who is an expert in the multi pairs stat-artitrage strategies is Manov. He made 10 place from the mql competition of the last year trading a basket of pairs. Also vgc has a moderate interest in the field too.
I am thinking about arbitrage in box options but this is a remote project.
This issue is the top of the top in FX stratefies and requires a huge amount of work.
vgc has written about this subject, I mean the basics maybe I could translate some of his work if he allows me.
I'm still learning how this site works/ is organized. It's a bit difficult to figure out where to insert myself so I'll just comment in this thread.
As a follow up to the last post, from my development environment which can work with COM, I'm now starting to use http://cran.r-project.org/other-software.html
(post chopped off) ...the R(D)-Com interface so far I'm impressed. It overcomes some of the problems with MTR4. Such as broken sessions whenever R encounters an error (a big problem), no feedback for error or formula evaluations, and ability to pass any R output back. I'm going to convert all my coe to use R(D)-Com over MT4R.
I'm not sure where to post this so I'll do this here:
I've been working on implementing SVMs but in getting a handle on R with time series analysis, I'm starting reading / working through the following book: "Data Mining with R - Learning With Case Studies." The book seems written from an application and beginning practitioner standpoint rather than as a theoretical or advanced math book. It covers a wide area of topics and shows how to implement them in R.
If you don't have (or can't find) the book but would like to work through the examples with me, send me a message and we'll work something out. Here is a review of the book that may be of interest:
http://www.r-bloggers.com/finally-a-practical-r-book-on-data-mining-data-mining-with-r-learning-with-case-studies-by-luis-torgo/
I looks great. Until now there are three main possibilities.
Matlab + MT4 datafeed : a library was given
R - MT4 data feed ?
Rapid Miner - MT4 Datata feed (Rapid miner is integrated with R)
I am on the third project.
John, what is the interface used to connect Rapid miner with MT4?
The simplest form is to use an indicator that will export the data in a CSV file
http://beathespread.com/file/view/364/indicator-exporting-values-of-ssa-cci-and-ivar-in-csv-file-for-use-in-rapid-miner
Then when we are in Rapid miner we can call the file by the operator read CSV. And anytime you press the run button it will read the updated CSV file. The CSV file is found in experts/files folder and in rapidminer you just have to precise the path.
As you know there is an integration tool between Rapid Miner and R. Rapid Miner can read R.
I had a lot of problems for real time data stream with QuandMod and R and I gave up.
The advantage of this is that by calling the iCustom function you can load in Rapid Miner any indicator you wish from Metatrader. And we have a lot of stuff by now.
Well this is not very user friendly but it is what I have.
After fighting with RapidMiner for the past few hours attempting to install the R extension, I think I have it worked out for R version 2.13.1. If anyone is having problems, see this post and change your environment variables to match what Sebastian has for his (allowing for differences with 64 bit if you're using that and your specific directory). I must say the documentation for installing the R extension is really hard to understand. Now where do I start? ;))
http://rapid-i.com/rapidforum/index.php/topic,3040.msg12094.html#msg12094
There are many algorithms for training Neural Networks.
The most popular is Back Propagation Algorithm. It requires “teacher”, ie special etalon signals for your Neural Network Output, and once you have those signals – your Neural Network will adapt to market, using those signals. But what if you don’t know anything about etalon signals and just have your indicator and want to have as much profit as possible ? What if we build some algorithm, which will adapt our network for profit ? There are few algorithms, able to help us, and the most well known are Monte Carlo (random search), Genetic Algorithm (evolutionary optimization search) and Particle Swarm Optimization (remember ants and bees?..). Monte Carlo is very slow, but it will eventually find some good weights for your Neural Network (in a year or two… or ten). Genetic Algorithm is much better, since it combines Monte Carlo method and Evolution method, this algorithm is proven to find solution (ie max profit in our case) pretty fast, but for Neural Network training (where you could have a lot of weights – parameters to train) the time is still major factor – training network of “10-5-1” architecture would take few days. So what about PSO? Particle Swarm Optimization was developed by people who were watching ants and bees achieving their goals together. In case of ants they built their homes pretty fast and coordinated, and bees had to search fields for their food (honey). People didn’t know how they do it, but it seemed very effective, so they modeled their behavior and created PSO. PSO generates species. Each specie is standalone worker (ant). The optimization search is done by all of the species across the Neural Network weights space. Species “communicate and coordinate” with each other, and that allows them to find best solution. Amount of species in PSO can be like 1000x smaller than in Genetic Algorithm, so adaptation speed is like 100-1000x faster.