Backtesting with data from CSV file

aharonzbaida's avatar

aharonzbaida since: 01 Jan 2015;

  14 Jan 2016, 23:16
Backtesting with data from CSV file

AlexanderRC said:

Spotware said:

Is tick data being downloaded from broker server to user's computer? if yes, where can i find them? if no, would like to know how reliable is tick data backtest in calgo. afaik, in mt4, tick data has to be downloaded locally so not sure if calgo does the same?

cAlgo automatically downloads tick data from Spotware's servers. Tick data is broker specific. Tick data is 100% reliable because it contains every tick. You can find tick data on your machine in the following directory:

C:\Users\%UserName%\AppData\Roaming\%BrokerName%-cAlgo\BacktestingCache\Ticks

For every date there is a separate file in folder named after the symbol. The files are in a proprietary format with scrambling (reversible encryption). The only value of looking at them is to check whether the specific date and symbol has been cached or not.

and it also seems like users need to be connected to the internet to run a backtest, why?

We plan to remove this restriction in the future. Thank you for your feedback.

Please add the ability to use user specified tick data from a CSV file at the same time.

Hi Alexander,

Spotware, and the community, does anyone have news about importing tick data into back tester? The files in the cache directory are not human readable format with .tdbc extension. Any suggestion on converting tick data to this format, or other method available to import historical tick data?

- Roni

Spotware's avatar

Spotware since: 23 Sep 2013;

  15 Jan 2016, 13:38

Dear aharonzbaida,

It's still not possible to backtest using custom tick data. We cannot provide you with an ETA of this feature. Altering cTrader/cAlgo files is prohibited as it is a violation of the EULA.

TRADERS FIRST™ Vote for your favorite features: https://ctrader.com/forum/suggestions
aharonzbaida's avatar

aharonzbaida since: 01 Jan 2015;

  15 Jan 2016, 14:15
RE:
Spotware said:

Dear aharonzbaida,

It's still not possible to backtest using custom tick data. We cannot provide you with an ETA of this feature. Altering cTrader/cAlgo files is prohibited as it is a violation of the EULA.


OK Thanks for reply. please keep us posted on this issue. keep up the good work Roni

Dinger since: 13 Feb 2015;

  24 Jan 2016, 10:32
RE: RE: RE:

Tcl DataBase Connectivity (TDBC)

good luck figuring out how to use that.

Also if you use wireshark you can see that calgo is downloading tick data from their website, but that tick data also happens to be in a weird format.

Example:

http://spotware-ticks.s3.amazonaws.com/cs/pepperstone-live/1/eurjpy/b/15/06/11.csv

If their data went far enough back it may actually be worth the time deciphering.

aharonzbaida said:

AlexanderRC said:

Spotware said:

Is tick data being downloaded from broker server to user's computer? if yes, where can i find them? if no, would like to know how reliable is tick data backtest in calgo. afaik, in mt4, tick data has to be downloaded locally so not sure if calgo does the same?

cAlgo automatically downloads tick data from Spotware's servers. Tick data is broker specific. Tick data is 100% reliable because it contains every tick. You can find tick data on your machine in the following directory:

C:\Users\%UserName%\AppData\Roaming\%BrokerName%-cAlgo\BacktestingCache\Ticks

For every date there is a separate file in folder named after the symbol. The files are in a proprietary format with scrambling (reversible encryption). The only value of looking at them is to check whether the specific date and symbol has been cached or not.

and it also seems like users need to be connected to the internet to run a backtest, why?

We plan to remove this restriction in the future. Thank you for your feedback.

Please add the ability to use user specified tick data from a CSV file at the same time.

Hi Alexander,

Spotware, and the community, does anyone have news about importing tick data into back tester? The files in the cache directory are not human readable format with .tdbc extension. Any suggestion on converting tick data to this format, or other method available to import historical tick data?

- Roni

aharonzbaida's avatar

aharonzbaida since: 01 Jan 2015;

  24 Jan 2016, 11:42
RE: RE: RE: RE:
Dinger said:

Tcl DataBase Connectivity (TDBC)

good luck figuring out how to use that.

Also if you use wireshark you can see that calgo is downloading tick data from their website, but that tick data also happens to be in a weird format.

Example:

http://spotware-ticks.s3.amazonaws.com/cs/pepperstone-live/1/eurjpy/b/15/06/11.csv

If their data went far enough back it may actually be worth the time deciphering.

 

aharonzbaida said:

AlexanderRC said:

Spotware said:

Is tick data being downloaded from broker server to user's computer? if yes, where can i find them? if no, would like to know how reliable is tick data backtest in calgo. afaik, in mt4, tick data has to be downloaded locally so not sure if calgo does the same?

cAlgo automatically downloads tick data from Spotware's servers. Tick data is broker specific. Tick data is 100% reliable because it contains every tick. You can find tick data on your machine in the following directory:

C:\Users\%UserName%\AppData\Roaming\%BrokerName%-cAlgo\BacktestingCache\Ticks

For every date there is a separate file in folder named after the symbol. The files are in a proprietary format with scrambling (reversible encryption). The only value of looking at them is to check whether the specific date and symbol has been cached or not.

and it also seems like users need to be connected to the internet to run a backtest, why?

We plan to remove this restriction in the future. Thank you for your feedback.

Please add the ability to use user specified tick data from a CSV file at the same time.

Hi Alexander,

Spotware, and the community, does anyone have news about importing tick data into back tester? The files in the cache directory are not human readable format with .tdbc extension. Any suggestion on converting tick data to this format, or other method available to import historical tick data?

- Roni

 


I am NOT trying to re engineer the system. I am confident that Spotware will realize the need, and sort out the broker concerns to allow this functionality. I would like to see a more sophisticated back tester that allows not only importation of large sets of tick data, but also functions such as dataset splicing for walk forward testing, data cleaning and normalization preprocessing, and Monte Carlo simulation along the lines of what Dr. Howard Bandy describes. Please vote so if you agree.

Dinger since: 13 Feb 2015;

  24 Jan 2016, 21:08
RE: RE: RE: RE: RE:

This post was removed by moderator.

ThirtysixD since: 07 Nov 2014;

  30 Jan 2016, 03:01

After loading my csv file when trying to start to backtest I get an error

Cannot read CSV file - IC Markets cAlgo
'Object reference not set to an instance of an object

Here is my csv file format
Date,Time,Open,High,Low,Close,Volume
2001.01.02,23:09:00,268.8,268.8,268.8,268.8,4
2001.01.02,23:12:00,268.8,268.8,268.8,268.8,4
2001.01.02,23:16:00,268.9,268.9,268.9,268.9,4
2001.01.02,23:38:00,268.8,268.8,268.8,268.8,4

Any ideas?

 

RhettG since: 10 Sep 2014;

  24 Jun 2016, 04:14
Backtesting with CSV data

Hi

I've loaded up data for 4 years on the DAX from a CSV file but there's no option to start the testing or to select dates according to the data I have. What do I do next?

davidp13 since: 06 May 2014;

  27 Jul 2016, 08:12

I have also selected the CSV file, but have not ability to choose the dates same as the last note from back in 2014. Why is this so? Am I doing something wrong?

davidp13 since: 06 May 2014;

  05 Aug 2016, 07:00
RE:

Admin pls respond. When I try to run Opt over the csv data it does not work. Gives an error. WHy?

davidp13 said:

I have also selected the CSV file, but have not ability to choose the dates same as the last note from back in 2014. Why is this so? Am I doing something wrong?