Multiple cTrader’s Running Same cBots?

Multiple cTrader’s Running Same cBots?

yeh I’m expecting notification of at least a partial refund for the outage will wait and see/contact them if don’t hear anything

Thanks for explaining your strategy and I don’t want to keep on about this as I know it is never possible to account for all the markets can throw at us (but this is not actually about the market) but even so with what you are doing what if the outage occurs when you’re sleeping? (low equity a thousand pounds or so ok can live with it but what about 10’s or 100’s of pounds equity?) I see NYC Servers offer a service to start up trading software after a restart (so umm isn’t that kinda admitting sh*t can happen even for them?) but I think this is only for MT - not possible with cTrader (and anyway starting cBots unattended with all their variables possibly in a mess due to the ’missing time’ spent in the outage/again depends on style of trading/cBot code would not be too cool)

Anyway I don’t know what the solution would be and as Panagiotis said there are many (most probaly all) running on a single VPS so I guess an outage and any potential loss caused by it would just be put down to experience and the overall P&L of trading as a whole... btw I did try running both VPS’s with same cBots at the same time and there were no issues when a trade  closed/triggerd  the next trade EXCEPT for the fact that each cBot placed a trade when it should have only been one (I use/depend on the TradeResult in the code to hold back further trades until initial has been registered succesful - obvously this result cannot (that I know of) be shared between the two cBots so I don’t reckon that idea is going to work for me anyway at least not for the current code but good to know it is ‘possible’

RE: RE: RE: RE: RE: RE:

xammo said:

yeh I’m expecting notification of at least a partial refund for the outage will wait and see/contact them if don’t hear anything

Thanks for explaining your strategy and I don’t want to keep on about this as I know it is never possible to account for all the markets can throw at us (but this is not actually about the market) but even so with what you are doing what if the outage occurs when you’re sleeping? (low equity a thousand pounds or so ok can live with it but what about 10’s or 100’s of pounds equity?) I see NYC Servers offer a service to start up trading software after a restart (so umm isn’t that kinda admitting sh*t can happen even for them?) but I think this is only for MT - not possible with cTrader (and anyway starting cBots unattended with all their variables possibly in a mess due to the ’missing time’ spent in the outage/again depends on style of trading/cBot code would not be too cool)

Anyway I don’t know what the solution would be and as Panagiotis said there are many (most probaly all) running on a single VPS so I guess an outage and any potential loss caused by it would just be put down to experience and the overall P&L of trading as a whole... btw I did try running both VPS’s with same cBots at the same time and there were no issues when a trade  closed/triggerd  the next trade EXCEPT for the fact that each cBot placed a trade when it should have only been one (I use/depend on the TradeResult in the code to hold back further trades until initial has been registered succesful - obvously this result cannot (that I know of) be shared between the two cBots so I don’t reckon that idea is going to work for me anyway at least not for the current code but good to know it is ‘possible’

No worries on asking questions - I'm happy to answer.

" but even so with what you are doing what if the outage occurs when you’re sleeping?"

My bots have certain time limitations I can configure. For example, I have it configured so no trades are placed about an hour before I typically go to bed. An hour before going to bed any trades I'm in should have at least turned a profit where I could have locked something in.Otherwise, I might tighten the SL a bit and then just let it manage the position overnight. But definitely no new positions will be opened.

In regards to your issue with the cBots bot placing orders, obviously one thing to do is check the Positions[] for the same symbol/timeframe/label just before opening a new one. All my bots have a uniquely encoded position label, so I can easily check that. It also allows me to open multiple positions of the same symbol across multiple time frames with the following:

_p = Positions.Find(_positionLabel);

You can narrow it down further via the API with:

_p = Positions.Find(_positionLabel, Symbol.Name);

_p = Positions.Find(_positionLabel, Symbol.Name, _p.TradeType);

But I have everything encoded in the label so it's less work the Find method has to do. :-)

RE: RE: RE: RE: RE: RE: RE:

Ah right ok so a slow and steady kind of cbot :) that you seem to intervene with quite a bit so I guess more of a semi-automated strategy - sounds good to be honest

yeh I’ve been at this (cTrader) quite a while now (2 or 3 years + and the same again with ProRealtime before that - never got into MT for some reason) and it took me a bit of time to get to grips with position labelling but got all the help I needed on here (as cryptic as it seemed at the time and still does! but have learnt a lot of C#!)

I prefer LINQ for position lookups and use;

Positions.Where(pos=>pos.Label == label);

then can include things like && pos.TradeType == TradeType.Buy or && pos.NetProfit > netProfit etc. to get exactlly what you’re after (yes I also include all the ‘vital info’ I need in the label) - also works great for ‘foreach’ statements and counting and detecting existing positions etc. but as I say I use/depend on a TradeOperation (sorry I said TradeResult earlier - always get mixed up on those two) off of an ExecuteMarketOrderAsync so it is not possile to ‘detect’  a position (by looking it up) until it has actually completed/impossible to do whilst the transaction is ‘flying in the air’ which is where the TradeOperation comes in - unfortunately the TradeOperation is cBot specific as far as I know so one cBot doesn’t know the other has already fired off a trade operation so they both do it/no way can detect it by counting positions etc. as they both will fire near simultaneaously/the position doesn’t exist at the time they trigger to trade

Also seems you’re using server side stops/limits set by your cBot so that will obviously ‘do the job’ managing the trade(s) if the lights go out on the VPS

I’ve dabbled with more techniques than I care to remember (have a folder stuffed full of all my previous attempts!) but at the moment I do not use any server side operations - everything is controlled by the cBot hence my desire to have as robust a solution as possible - I realise as I am writing this I said above I have tried to build things to take into account issues but here I am admitting I don’t even use server side operations to ‘deal with things’ if the cBot goes offline for whatever reason but.. well...that is to do with the style of trading I am doing which is not particularly fast moving and I could/have used server side operations to deal with things but for reasons that escape me now I decided to have 100% control within the cBot itself as my strategy depends on new trades triggering on others closing (could be hours/days between trades) so I can survive 9/10 if the VPS goes offline unless like this time it just so happens a trade is very near to closing/subsequent trades should fire (I guess I am trying to reduce ‘sods law’ haha ;)

Anyway thanks for the feedback all helps in broadening the mind! :)

RE:

OK.... so I setup and have been using a NYC Server (Windows Server 2012 R2...! which I am typing this on now) for the last week or two and running a few bots live and whilst initially everything seemed ok I am now experiencing relatively frequent broker disconnects which seem to happen when I connect to the VPS (cTrader goes into a bit of a fit and the GUI gets messed up and is not showing properly the list of bots in automate and the broker connect is attempting then after 10-20 seconds things spring back into life) but other times I am only aware of it as I have not received any emails from the bots running that trigger trades every few hours or so on average and when I logon to the VPS I see the previous mentioned ‘fit’ and the broker connection eventually made then a handful of trades trigger that should have triggered some time ago! very frustrating :(

I am fed up with it so decided to write the following code in an attempt to catch the issue but every time the disconnect happens the bot (below) has stopped and no email is sent (email setup is 100% ok/the other bots running fire off emails no problems) so please advise what is wrong with the code or how I can achieve the desired results of sending emails every time a connect/disconnect happens

But also just to add Panagiotis with all due respect I think you and I both know that a single point of failure in any system is not ‘good enough’ and using a recommended VPS is still a single point of failure no matter which way you look at it - I have seen other posts going back 5+ years that are requesting greater fault tolerance and resilience of cTrader such as being able to autostart bots if a machine restarts unexpectedly/system crash - I totally appreciate this is no doubt complicated but I also do not believe this can be passed off by saying it has anything whatsoever to do with ‘no system can run 24/7 without any intervention’ - this is is not what we are talking about here and I really think Spotware need to start putting some effort into making their software (up for an award I think I saw - hope that goes well!) more resilient (for example why isn’t there already a built in tick box in the settings to send an email on server connect/disconnect?) and what about developing the ability I already spoke about where the same bot can run concurrently on different machines (multiple machines managing the same code/positions at the same time - I am  sure that would win Spotware many awards! :)

Thanks

MaxT

using System;
using cAlgo.API;

namespace cAlgo.Robots
{
    [Robot(TimeZone = TimeZones.UTC, AccessRights = AccessRights.None)]
    public class ServerState : Robot
    {
        protected override void OnStart()
        {
            Server.Connected += OnServerConnected;
            Server.Disconnected += OnServerDisconnected;
        }

        protected  void OnServerConnected()
        {
            SendEmail("Connected "+Convert.ToString( Environment.MachineName),Convert.ToString(Server.TimeInUtc));
        }

        protected void OnServerDisconnected()
        {
            SendEmail("Disconnected " + Convert.ToString(Environment.MachineName), Convert.ToString(Server.TimeInUtc));
        }

        private void SendEmail(string subject, string body)
        {
            Notifications.SendEmail("abc@gmail.com", "xyz@outlook.com", Account.IsLive ? "[LIVE] " + subject : "[DEMO] "  + subject, body);
        }

        protected override void OnStop()
        {
            Server.Connected -= OnServerConnected;
            Server.Disconnected -= OnServerDisconnected;
        }
    }
}

Hi xammo,

I am fed up with it so decided to write the following code in an attempt to catch the issue but every time the disconnect happens the bot (below) has stopped and no email is sent (email setup is 100% ok/the other bots running fire off emails no problems) so

The events provided by the cBot are triggered only when the server disconnects you for any reason. When disconnections occur for any other reason beyond our control (e.g. bad connection, poor server performance) then these events will not be triggered.  

please advise what is wrong with the code or how I can achieve the desired results of sending emails every time a connect/disconnect happens

Based on the above, probably the events were not triggered at all. Also if your disconnection is caused by a general internet disconnection, obviously no emails will be sent

But also just to add Panagiotis with all due respect I think you and I both know that a single point of failure in any system is not ‘good enough’ and using a recommended VPS is still a single point of failure no matter which way you look at it

We recommend this as the first step to take for your cBot to run 24/7. Frequent infrastructure outages is something you need to check before proceeding with a software solution to the problem. Disconnections every some hours should not be the case. Is your VPS provider reliable? Where is your VPS located? What is the ping time? Do your machine specs meet the recommended requirements? Can it handle your cBot execution efficiently? In any case, if you need fault tolerance, this is something you need to develop it yourself, since a cBot is your software.

 I have seen other posts going back 5+ years that are requesting greater fault tolerance and resilience of cTrader such as being able to autostart bots if a machine restarts unexpectedly/system crash

Here it is

 

 I really think Spotware need to start putting some effort into making their software (up for an award I think I saw - hope that goes well!) more resilient (for example why isn’t there already a built in tick box in the settings to send an email on server connect/disconnect?)

Our part of the software is fault tolerant. Our servers are up and running for 10 years now and we had no downtime yet.

(for example why isn’t there already a built in tick box in the settings to send an email on server connect/disconnect?)

As explained above, we cannot know if there is any issue on your side. You need to handle these cases yourself.   

 what about developing the ability I already spoke about where the same bot can run concurrently on different machines (multiple machines managing the same code/positions at the same time - I am  sure that would win Spotware many awards! :)

The same cBot can run concurrently on different machines. Nothing prevents you from doing so. But you need to handle the concurrency issues yourself, since it is your logic that is being executed. This is not something cTrader can handle for you, cTrader is just the platform for your strategy to be executed on. It is like asking from a multicore CPU to do the parallel programming for you. This is not possible. If you need concurrency, you need to write the cBot code in a way that it can run concurrently on many machines. 

Best Regards,

Panagiotis 

Join us on Telegram

Head of Community Management at cTrader
RE:

Hi Panagiotis

Many thanks as always for an extremely quick and very thorough reply! I wrote my reply earlier but on submit it seems not to have actually posted and just took me to the homepage (something I forgot happens when posting on this forum/I usually always copy and paste the whole reply before hitting submit just in case but didn’t bother this time more fool me!) but I thought I’d go and do somethibg else instead and give the changes made some time to ‘work’ and hopefully have more to report back 

I am fed up with it so decided to write the following code in an attempt to catch the issue but every time the disconnect happens the bot (below) has stopped and no email is sent (email setup is 100% ok/the other bots running fire off emails no problems) so

The events provided by the cBot are triggered only when the server disconnects you for any reason. When disconnections occur for any other reason beyond our control (e.g. bad connection, poor server performance) then these events will not be triggered.

Ok understood thank you for the explanation (I would use other methods external of cTrader to detect connectivity/machine performance issues anyway)  but obviously there were  no connectivity issues as I was connecting to the VPS at the time

please advise what is wrong with the code or how I can achieve the desired results of sending emails every time a connect/disconnect happens

Based on the above, probably the events were not triggered at all. Also if your disconnection is caused by a general internet disconnection, obviously no emails will be sent

I think they were triggered as the code bombed out and stopped the bot also as per my edit post above I realised I had mistakenly left the ‘protected’ instead of changing to ‘private’ for the two methods

Yes I do of course understand no internet connectivity means no emails can be sent but as stated it is a VPS I am connecting too so no connectivity issues otherwise I would be unable to connect and the other bots are firing out emails just fine

But also just to add Panagiotis with all due respect I think you and I both know that a single point of failure in any system is not ‘good enough’ and using a recommended VPS is still a single point of failure no matter which way you look at it

We recommend this as the first step to take for your cBot to run 24/7. Frequent infrastructure outages is something you need to check before proceeding with a software solution to the problem. Disconnections every some hours should not be the case. Is your VPS provider reliable? Where is your VPS located? What is the ping time? Do your machine specs meet the recommended requirements? Can it handle your cBot execution efficiently? In any case, if you need fault tolerance, this is something you need to develop it yourself, since a cBot is your software.

As previously stated it is a NYC Server VPS and there does not seem to be any strain on the CPU/RAM with the currently three bots running so I assume all the remaining questions are null and void apart from the statement about fault tolerance which yes I understand

 I have seen other posts going back 5+ years that are requesting greater fault tolerance and resilience of cTrader such as being able to autostart bots if a machine restarts unexpectedly/system crash

Here it is

 

I take my hat off to you sir!!! I do recall seeing this in a relatively recent update (last year or so or maybe bit longer?)  but as at the time my thinking was more along the lines of multiple instances did not feel the need for it and to be honest totally forgot about it! Thank you very much for pointing it out - I set it up immediately after reading  - restarted  cTrader with the bots still running and yes! they all started immediately - superb and obviously a perfect combination with NYC’s ‘widget’ to allow for autologon and autostartup on startup of  cTrader - that pretty much solves unexpected machine restarts! Thank you very much!

 I really think Spotware need to start putting some effort into making their software (up for an award I think I saw - hope that goes well!) more resilient (for example why isn’t there already a built in tick box in the settings to send an email on server connect/disconnect?)

Our part of the software is fault tolerant. Our servers are up and running for 10 years now and we had no downtime yet.

Yes I was not referring to your backend infrastructure and honestly I do not say this lightly congratulations on such an extremely impressive track record that is quite amazing and I am sure without a shadow of a doubt a great many factors contributed to achieving and are continuing to contribute in maintaining such a resilient and robust system which is the wisdom and knowledge I am basically suggesting would be great to extend out over to the front end to your clients that are using cTrader for algo’s (not manual trading/semi-automated/web based etc.but for full time algo clients looking for the most robust and resilient setup they can achieve - it would set you leagues apart from others) 

(for example why isn’t there already a built in tick box in the settings to send an email on server connect/disconnect?)

As explained above, we cannot know if there is any issue on your side. You need to handle these cases yourself.   

Yes understood but as you also explained above you are able to detect server disconnects initiated  by the broker so why not alert the client to these disconnects as the norm/switchable setting?

 what about developing the ability I already spoke about where the same bot can run concurrently on different machines (multiple machines managing the same code/positions at the same time - I am  sure that would win Spotware many awards! :)

The same cBot can run concurrently on different machines. Nothing prevents you from doing so. But you need to handle the concurrency issues yourself, since it is your logic that is being executed. This is not something cTrader can handle for you, cTrader is just the platform for your strategy to be executed on. It is like asking from a multicore CPU to do the parallel programming for you. This is not possible. If you need concurrency, you need to write the cBot code in a way that it can run concurrently on many machines. 

Yes of course I am not looking for the CPU to do the programming for me I understand what you are saying but I was thinking/hoping there might be something behind the scenes perhaps even between spotware and the broker that might help in this regard but sounds like nothing along those lines exist

I have tried to think how to rewrite my code to achieve anything that might work but am at a loss as previously stated my code relies on the TradeOperation (or TradeResult) to manage the trades and as  bots cannot ‘talk to each other’ I cannot see how it is possible other than a couple of ideas have been to write a file every tick to a shared location that is time stamped and read  by the paired  bots on different machines but other than the logic I would think file read/write/sync latency would basically wreak havoc  - another was along the lines of the bots pinging each other in a master/slave scenario and brief searches brought up TCP transmitter and listener as being available in C# but even whether it is anything that would help or even does what I think it does I don’t know and can see many hours being consumed just trying to work out what it is and how to set it up/use it...!

Anyway on a final note and now being some hours later having switched cTrader to single instance mode I have not experienced a single disconnect... not one... hmmm

Cheers

xammo

RE: RE:

Ok something very odd has just happened... since writing the last message I was busy with an online meeting for two hours - after the meeting finished I logged onto the NYC VPS to check everything was ok

I wasn’t quite sure what I was seeing as the screen seemed to be glitching and refreshing badly all messed up but as it ‘came right’ cTrader was starting up and the bots were firing up and the server connectivity attempting to connect which it did eventually and everything seemed to settle down but obviously something had been very very wrong...

I checked my emails to see what had been received and am attaching the screenshot from my phone

(Sorry am sending from my phone - that image looks huge/cannot see how to resize) the coloured circles showing green for the emails which I have configured the bots themselves to send on order execution then the red one which is ‘Liberty’ (NYC Servers ‘widget’) notifying me that cTrader had become unresponsive at about quarter to nine (am in Asia/Thailand/is Thai time) and was attempting to restart it but you can see the amber circled email is me logging onto the server at quarter past nine which is when I saw cTrader frantically  starting up so the restart of the unresponsive cTrader was obviously unable to happen without a user actually being logged on so unfortunately that doesn’t seem to have worked well

Then you can see the next green circled email which is one of the bots executing a trade that obviously should have happened during the time cTrader had crashed...

Ok I thought so things happen and this is all new territory and what the h*ll just happened etc. as cTrader suddenly disappeared and began restarting... wtf! Ok it’s started up again wtf is going on... woah it’s restarting again!

after this third restart I stopped the ServerState bot (code above) thinking it might be something to do with it but still it restarted two more times which is when I set cTrader back to multi instance and didn’t even restart it just left it like that and it hasn’t restarted in near on an hour now/everything seems stable again...

hmmm... not good... enough for today though just wanted to report back what happened

RE: RE: RE:

Hi @Xammo:

I have been using NYC Servers for years and have never experienced any of the issues you're reporting. I'm using the standard cTrader set up and typically have 2-3 instances of cTrader running with multiple bot-instances of each executing. Mine don't send emails or have any of those event handlers you're using though.

What I would suggest is having them wipe down and re-image your VM so you start from a clean slate.

Also, if you submit a support ticket, as for "Nick Esposito" as he's the person who I have been dealing with the entire time.

Hope you figure out what's happening.

RE: RE: RE: RE:

Hi firemyst

Thanks for the reply/I thought I would leave it a week or two before replying to see how the changes I made worked out

I am not experiencing lots of issues only two but they are major issues obviously - I do not think it is anything to do with the server build/all seems fine on that side

I removed a print statement (that I could toggle on/off anyway) and the streamwriter file writes from within my code and basically stripped it down to the absolute bare miniumum code I can manage to still actually do what I want it to do! Overall I would say this seems to have sorted out the issues although they are not completely gone - I am no longer getting server disconnects (although I did once and to me it seems it is happening when I try to recconnect to the server via my iPad using MS RDP when I have not properly disconnected from the server and swapped to another app on the iPad then 10-15 minutes later switch back to the RDP app and it tries to reconnect - this seems to send the server into a bit of a spin and cTrader server broker connection is attempting to connect - obviously now I always disconnect properly from the server before swapping to another app and this seems to be working!)

The other issue is with the NYC Servers Liberty ‘widget’ which I guess you don’t use as you have multple instances of cTrader running - it seems to be a little too aggressive in determining an app has crashed when it is actually just ‘having a moment’ but it treats it as a crash and restarts it - I also added the taskmanager to the widget as I always have that open to monitor realtime CPU/RAM and this too has ‘crashed and restarted’ a couple of times by the widget so yeh it seems to be too agressive/might contact NYC if keeps happening

Overall though happy to report things are working ok now but I am still not satisfied this is a robust enough solution to build up (hopefully!) to be trading 10’s/100’s thousands of $$’s and sleeping easy at night - I still wish more focus could be put on providing increased redundancy for 100% automated algos but I guess it is down to the trader themselves to work that bit out...

cheers

xammo