Developers

 

Streamly Developers Page

Start Processing Streams Instantly with Streamly

In less than 5 minutes, you can be up and running with Streamly, using your analytics applications of choice. Streamly needs no installations, configurations, or integration. Streamly gets you started in the fastest time possible.

Getting Started

 

/
Register on the Streamly Platform:

The registration process is as straightforward as any other. Simply input your email address and password and you are ready to go. Behind the scene, your complete workspace is created (Zeppelin, Kibana, Kafka, etc.).

/
Select the tools required for your pipeline:

Streamly integrates all the basic tools you need for your processing pipeline. Identify each of these tools and create the necessary resources (e.g. topics, keyspaces, tables, indexes, notebooks, dashboards, etc.).

/
Start Processing Streams:

Upload your Spark application and start it. You can directly see the logs flowing to Elasticsearch and available in Kibana. You can also see your application metrics in your dedicated Spark UI. Everything is automated. No manual configuration.

/
Tutorials

Given Streamly is powered by open-source tools, learning Streamly simply amounts to learning the tools relevant to your needs. And, if you already know your tools, then you are ready to start using Streamly right now. Nonetheless, we have prepared a few sample Spark applications that you can find in our GitHub repository.

/
Getting Assistance

If you have any questions or problems when using Streamly, please make sure you check our frequently asked questions. If you do not find the answer to your question in the FAQ, you can post it in the mailing list. And, of course, you can always contact us, especially if your question is confidential. We would be delighted to help where possible.

/
Learning materials

Below are links to materials to help get you up to speed with Stream processing:

KafkaSpark 1, 2, 3CassandraMQTTMQTT ClientsZeppelinElasticsearchLogstashKibana 1, 2

We’re working on expanding this list; Let us know if you are missing something.

Streamly Developer Tools

/

Apache Spark:

This is the central processing platform onto which stream applications are deployed. These applications then read data from Kafka or MQTT, using dedicated and high-performance connectors. Numerous built-in technologies can be used to facilitate processing, such as Spark Streaming, MLib, GraphX, Tensorflow, Logstash, etc.

/

Kafka:

is tightly integrated as a messaging conduit to establish a reliable and asynchronous communication channel between the data sources and the remainder of the solution. Example data sources are sensors, IoT devices, logs, click streams, tweets, blockchain events, vehicle data, healthcare devices, sensor networks, etc.

/

Electricsearch:

Processed data can be written to Electricsearch so it can be searched. This tool is deployed in a reliable and highly-available architecture and tuned for high performance. A strong security plugin enables multi-tenancy and guarantees that each user’s data is properly siloed. Stream processing logs are also stored in Elasticsearch.

/

Kibana:

is the visualization tool for every Elasticsearch data. This includes logs, metrics, or any other processing results. You create Kibana graphs that you group into dashboards. Multi-tenancy is achieved by storing each user’s data in separate indices in Elasticserach, where they are further secured using a strong access control mechanism.

/

EMQTT:

is also integrated as a messaging conduit to establish reliable and an asynchronous channel between the data sources and the remainder of the solution. MQTT is often used by lightweight devices such as sensors or IoT devices. Topics are created and secured in the Streamly dashboard. EMQTT is the chosen implementation and is deployed as a cluster.

/

Cassandra:

a massive scalable and highly-available NoSQL database that can be used by the stream processing application as storage. Zeppelin tasks can also read and write to Cassandra. The deployment currently includes two datacenters. Keyspaces and tables are created in the Streamly dashboard and are automatically secured.

/

Zeppelin:

A web-based notebook that enables interactive data analytics on your data streams. The necessary interpreters are already installed and configured to support a multi-user environment. These interpreters allow access to authorized keyspaces, tables, topics, and indices. Zeppelin allows analysts to discover insights in their data streams.

/

Streamly:

The invisible Streamly glue that ensures strict separation between customers. It also provides a uniform interface that guarantees a consistent user experience across these technologies and boosts productivity. Above all, the Streamly glue automates such mundane but vital tasks as platform monitoring, operation, healing, rolling upgrades, and backup.


Benefits to Developers

/
Open Streams

You can take advantage of Streamly’s open streams which are available to all registered users. Example of open streams currently available to you are: Bitcoin, Ethereum, and Apache Logs. Stay tuned for more exciting streams.

/
Get Started Fast

No installations, integration, or fine-tuning means you get started fast and can focus on your core business. Most tools in a stream processing solution are non-trivial to operate. Streamly helps you avoid this hassle.

/
Affordable

Real-time data processing is expensive, especially when combining different cloud vendors. Our one-stop-shop approach allows us to control and cut costs, making stream processing more accessible to you.

/
High Availability

Streamly is built on a modern architecture, with each tool deployed under supervision and in a high availability mode. Examples are multiple datacenters for Cassandra as well as high availability mode for Apache Spark.

/
High Performance

We continuously do performance testing and tuning so you do not have to. While each tool of the architecture performs well in isolation, integration is a notorious source of bottlenecks. We address these bottlenecks head on.

/
Open Source

Streamly is based on proven state-of-the-art open-source components. As a result, not only do you avoid being locked into a specific vendor, but you also build important skills that you can use in your career and in your other projects.