beathespread

Check this artcilce it is quite interesting by offering an independant validation tool for trading system.

Here I can just summarize the common basic tools for validation of trading systems.

-Sinple Back Testing with or without Out of Sample Validation

-Walk Forward Testing

-Monte Carlo analysis

-Bootstrapping procedures 

Here on this article a different paradigm is offered related with the Robust Control. It is worth to have a look at it.  

Here is a video of the author

http://www.myspace.com/video/zerofrancisco/robust-control-paradigm-for-trading-of-equities/23615453

Guys you could really  watch this video the week - end, the video and sound quality is poor but content is king. I am analysing it.

The real problem is that the application of data mining on trading may be a wrong paradigm.

Check was an experienced trader has to say on the topic.

http://tradersplace.net/forum/thread/60/monte-carlo-analysis/

So I can add:

data - mining + back - testing  =  tools for fools.

In fact you can come out in the back - test pretty easily with nice looking equity curves.

You can ask why a simple moving average cross over system with extensive search (brute force search) on all parameters has to be outperformed by any complex data mining algorythm?

Theoretically both approaches should work.

The problem with the market is that it is self reacting to itself. If the market was a mere physical process every data mining routine would work. But the market reacts to itself. That is why HFT was so profitable there, the first players in their time horizon were faced to a market that did not react to their strategies. And by doing so it was regarding to them a physical process that can be predicted. 

This is a simple question but the answer is not obvious.

Another remark is also very intereesting

" Backtesting is the method of choice for destroying your career and your firm, if you own it. Backtesting is toxic to quants. there is only one possibly useful alternative. backtest once with a well defined prior."  Richard Michaud

I think the robust control is the way to go. In that way everything mathces into mucch more complex paradigm. Data - mining and back - testing needed to be related with robust control. Otherwize it is toxic.