Understanding the Problem and Stack ( Litecoin Inference )

 I decided to rip off the bandaid and use my Python skills to do some Data Science and retire R, as it really wasn't serving me to learn neural networks. The first thing I had to do was to figure out all the Python libraries and this is what I settled on:  a client library for Twitter, a current stock data library, and a stock technical analysis library. After that, I found a great open API for NLP sentiment analysis. The next library I needed was data storage and manipulation. I used a db library to store the data and, finally, Numpy and Panda to manipulate the data while experimenting. 

This is what I built to solve the problem. 

I took around 1,000 tweets daily with the search term Litecoin and did sentiment analysis on them to find out of the 42 tweets per hour how many of those are positive, negative, or neutral, and stored that in a MySQL database table. 

Next, I took the stock data from Yahoo Finance and ran it through technical analysis software, looking for candle stick graph patterns like three white nights. I saved all the standard stock data, like open price, closing price, and volume, along with the five candle stick patterns I was looking for, into a table with a shared column name for the timestamp between the sentiment and stock tables. 

I would build the model based on this data selecting the most relevant features. 

The following are the libraries I used to collect and build the model. 

The Python Stack:

Tensorflow: A full features ML / AI stack from Google to build and run models in Python. 

Transformers: A open ML API from Huggingface, an Open AI company, with the open twitter-roberta-base-sentiment-latest model and API to perform sentiment analysis. 

Tweepy: Twitter client for Python that allows searches via API 2v ( worked until Mid June ) 

Yahoo Finance: Solid stock data from Yahoo Finance

Ta-lib formwork: Technical analysis framework that took some compiling but worked great https://ta-lib.org/ something to find candle stick patterns in stock data. 

Pymysql/MariaDB:  libraries used to work with my Marainda DB database. 

Pandas / Numpy for data and ETL handling. 

The biggest challenge was to get tensor flow working on my ARM Kubernetes hardware. I built my stack as docker containers and used docker build for container building, and then I uploaded my containers to Dockerhub in a private repo. Next, we will get into my ARM K3s Kubernetes cluster, the platform I chose for this project.  


Comments

Post a Comment

Popular posts from this blog

Litecoin Data Collection and saved data features

Legal Disclaimer:  The Content is for informational purposes only. You should not construe such information or other material as legal, tax, investment, financial, or other advice . Nothing contained on our Site constitutes a solicitation, recommendation, endorsement, or offer by myself or any third-party service provider to buy or sell any securities or other financial instruments in this or in any other jurisdiction in which such solicitation or offer would be unlawful under the securities laws of such jurisdiction. I have already talked about the two scripts, one that pulls the financial data from Yahoo Finance hourly, then looks for the TA signals in that data. The other script searches one thousand tweets daily, forty-two tweets every hour for Litecoin, and uses hugging face to analyze sentiment.  Let's talk about the data features we wanted to save and the tables we kept them in.  Transactions: We collect the following metrics open, high, low, close, adj close, volu...
Read more

CodeLlama for DevOps massive productivity hack.

Image
Codellama LLM for DevOps Using the OpenWebUI  Well, it happened. Thanks to quantized models and the specs of a Raspberry Pi, it is possible ( but not advisable) to get LLM working on a Raspberry Pi's 4B 8GB or RAM. I did some testing and got about 0.5 tokens per second on my Raspberry Pi. I will be getting a Pi 5 ( completed doubled the speed to just over 1tps). I ran it on My MacBook M2, and it was closer to 27 tokens per second, much faster.   Codellama 7B/13B is a game changer for DevOps Productivity as it can do everything from Switch to configuration examples, Ansible playbooks, Terraform, and Python code. I was able to generate solid working examples of all of the above.  Still, with it being very slow, it's helpful because you can ask it to run and get an example of code or configuration. Come back in 10-20 minutes. It has it. You can do it while in a meeting or working on something else.  I had some doubts, like it was talking to a backend or something, ...
Read more

DevOps LLM shootout TLDR

  DevOps LLM shootout TLDR The experiment is to take four everyday DevOps tasks and test them with the open and free-accessible LLMs available to compare them and see which solution is preferred based on several factors for DevOps Engineers.  We will rate them by the following: Correctness of solution Explaining the Answer Elegance of results A quick note on correctness. Things can be less than 100% accurate, but give the user the correct format, fields, examples, and explanations to fill in the blanks. We will score them 1-10 on each with a max score of 120 and share the results. We used ChatGBT 3.5 Free, Google Gemini, and Meta CodeLlama 13B models to ask the following prompts to compare here are the prompts: Write Ansible Playbook to install httpd, copy a template from a file, ensure the service runs, and post a simple Hello World webpage. Create terraform for an EKS cluster of 5 nodes using terraform with t3.medium nodes. Write a Kubernetes manifest for Minecraft wit...
Read more