At this point I dove into the Peltarion platform headfirst and hoped for the best. I uploaded the data, saved it and used it in a new experiment. I selected everything but the date as the input and the “Next day positive” as the target. The only thing in the modelling view which I understood at the time was the “Patience (epochs)” parameter which was set to 5, so I bumped it up to 20 to let my model run a bit longer.
To my surprise, the model was able to achieve a roughly 52% accuracy in predicting the price the next day. But fear not, I did not manage to get the model right the first time. The reason for this high accuracy is because 53.9% of the total days were positive, so the model actually performed worse than only guessing TRUE every day.
At this point (and a lot of points going forward) I noticed that the model wasn’t really learning anything. My limited knowledge about ML told me that a graph like this should look like a log(x) graph: steep in the beginning, and then flattening out after a while. My graph was going sideways the whole way. It looked as if it was just blindly guessing until it got something right.
So, I started testing some theories that I had for improving my model. My first theory was that the model could not handle that many inputs, but that turned out to be false, and the model performed worse with fewer inputs. My second theory was that I was using the wrong problem type, so I changed it from tabular classification to tabular regression. I wasn’t quite sure how this affected my model, since I couldn’t see any obvious changes in the modelling view, but the model performed better this time. It managed to get an accuracy of 54.2%, which is better than guessing true every time.
From here on I played with different settings, trying to learn what they all did. I discovered the dropout ratio, and it turned out that my model got better with a lower value. I discovered overfitting and underfitting. I discovered class weights, which I should have been using all along since my target class is not evenly split. None of these tweaks resulted in any substantial improvement in the model, and it still looked fundamentally flawed since it wasn’t really learning anything, the flat curve was just moving a bit higher or a bit lower on the Y-axis and getting random higher peaks.