3 Reasons to Run Your Business Intelligence in the Cloud

With hundreds of thousand users now working on different cloud applications like Google Apps for business, Salesforce’s CRM, and Quickbooks online for accounting, it is clear that the cloud is the way of the future.  If your business analytics solution has not already headed skyward, it is worth considering the move.  Here are three reasons to run your business intelligence solutions, ie business dashboards, analytics, and KPIs, on the cloud.

Accessibility: Available On Any Device

Cloud technologies are moving towards responsive designs which enable dashboards to automatically reorganize information based on screen size. New cloud technologies adjust reports to present the most meaningful information based on the size of the device. Due to faster internet speeds, live analytics database connections allow for real time or near real time updates to business dashboards and analytics even on small devices. 

Since most of the computing power is on the cloud servers, the devices only require a simple browser to display the visualization or reports.  Cloud providers enable developers to support multiple devices, screen sizes and operating systems, in some cases even while offline. What is better than having your business metrics and info wherever and whenever you need them?

No Need for IT Resources to Maintain Massive Hardware Infrastructures

This is one of cloud technology’s biggest pluses. Cloud BI allow businesses to focus on running and managing their core offerings while leveraging applications and letting the service providers take care IT and the related resources needed to operate and maintain a technological infrastructure.  

Rather than draining resources on hardware maintenance, operating systems, networks, backup, storage, load balancing, and on-site data storage, businesses can instead shift their focus to the productive use of their data.  Google Cloud Platform, for example, provides offerings ranging from Platform-as-a-Services (PaaS) to Software-as-a-Service (SaaS), which encompass database, storage, and security management solutions, to name a few.

Access to Continuous Improvement and New Features

In order to make updates to non-cloud technologies, you need to take the application offline, install the patch or service pack, and finally test to make sure the update integrated correctly.  Updates on the cloud are a breeze in comparison. In fact, with the cloud’s continuous updates and improvements on the backend, product upgrades or fixes can be rolled out easily without huge interruptions or additional software purchases.  Continuous cloud updates translate to an enhanced user experience.  Google App Engine, for example, allows developers to push out new versions with a single click, providing users with instant updates the next time they use the service or application.  

There are many more benefits of running your analytics running via cloud technology. What benefits were you surprised by?

Big Data Analytics: Marketing Came First. Why Engineering Is Next.

Big data solutions have made it.  It is clear that business intelligence and advanced analytics solutions have been embraced by the market and that adoption and innovation continue their upward trajectory.  Many of the first companies to provide big data solutions were marketing-focused companies specializing in predictive analytics.  As this market matures and marketing companies find their niches, the conversation and question naturally moves to “who is next?”.  What other enterprise group is ready to evolve and innovate big data analytics?

It does not appear to be sales despite early advancements in the analytics space. Perhaps because sales analytics investments were made early on and the benefits from those initial investments are still being reaped.

Customer service looks to be a likely candidate however budgets do tend to be tight for this group.

Our educated prediction here at Bimotics is that the next boom in big data analytics will be in the engineering space.  Here is why:

Systems or application logs: in order to provide real time monitoring, tech support and proactive maintenance, applications generate millions of rows of data, describing all kinds of processes.

Application Usage Analytics: customer experience and human factors are based on usage analytics. This means applications capture the flow and interactions of users in terms of features they use, time on the app or pages, and patterns of usage.

Metadata Management: Data traceability is of the utmost importance and "data about data", ie metadata, is a huge arena. Payload of data transfer is also considered in this category where sometimes the metadata generates more data than the payload itself.

Cohort Analysis: By developing A/B testing, engineers can improve conversion ratios or better UX / IU to increase usage. Cohort analysis uses two groups as a benchmark. This generalization of groups requires more data storage.

The big benefit of engineering-related data analytics is that insights and performance directly impact the products and services which are core to the business. Understanding how customers use your application in particular and what makes them tick can make product planning and prioritizing features more methodical and effective. This is why we believe that engineering analytics will be the next big wave in the sea of Big Data Analytics.

Building an Online Data Warehouse Part 3

Visualizing the Data

A picture is worth a thousand words.  No doubt, a well-crafted chart from sound data is similarly impactful.  An armful of those meaningful charts can lead to those a ha! “good to know!” moments that give your business that critical and much needed edge.

In our last blog, we explained that although marvin.'s own visualization engine is still in development, its analytical engine can currently be used for analysis by writing queries within the database. Soon, through marvin.’s user interface, or by leveraging other Google Technology Partners like Tableau or BIME, you will be able to create charts and visualizations on top of this database.  Doing so can help make sense of your data, identify patterns and trends, create reports, make measurements and ultimately help you learn and make decisions on how to improve your business- an ultimate cloud BI solution.

Currently, marvin. allows users to  configure databases in a way that is friendly to visualization engines available on the market. Databases can be set up to help data visualization engines recognize fields like text versus numbers versus dates and other options that help provide  meaningful insight. Ultimately, this visualized data can be combined to create dashboards where business users can see a tapestry of charts  rather than  one piece of data at a time.  Users will be able to define multiple dashboards, multiple charts, and multiple filters-- all based on their data.  

Visualizations conbimed with an online data warehouse open up a whole new world of business insight by allowing users to recognize and identify patterns that are almost impossible to see just by looking at numbers.  Dashboards with a whole host of these visualizations can provide you with critical information and rare insight about your business that you never had access to before.

Building an Online Data Warehouse Part 2

Google Cloud Platform is a great option for companies looking to build a Data Warehouse on the cloud.  As described in Part 1, you have two options: hire a developer to code and integrate on top of those great APIs, or use tools like marvin. to help you automate some of the processes that are required to build an online data warehouse. In Part 2 and 3,  we will discuss the most valuable sections of a Data Warehouse, Analyzing and Visualizing your Data.

Analyzing the Data

marvin.’s super powers come from Google BigQuery API, an analytical database on the cloud. marvin. automates the process of inserting and storing data uploaded in the previous step. Together, Google BigQuery and marvin. are the analytical engine for your data warehouse. Your new Data Warehouse is limitless in terms of size and analytical computing power. Even better, there is nothing to code, no servers to manage, and no hard drives to setup. To learn more check out our blog:  Google BigQuery: 7 fascinating facts.

With marvin., you create datasets, make tables that host data, and preview that data in a table format. The data is no longer raw. Instead, your precious data is stored like in a huge spreadsheet that can be analyzed very quickly. Think billions of records in seconds. Think Big Data. Think cloud bi.

What if your business only has a few thousand records? No worries. marvin. will manage small data just as well. After all, a small business does not need to analyze a billion records of its data to make informed decisions and leverage insight to compete with other businesses.

Your new analytical engine also allows users to interact with simple SQL statements. If your team does not know SQL, it is also compatible with multiple visualization engines like Tableau and BIME, popular programs that present your data graphically. Visualizing the data is the last piece of the 4 steps of the simplified process.  We will explore data visualization in Part 3.   

Building an Online Data Warehouse Part 1

Google Cloud Platform is a great option for companies looking to build a Data Warehouse on the cloud, with plenty of APIs from which to build.  Currently, the number of APIs which you can choose from to connect your applications to the cloud is growing by the day.  As of now, you have two options: hire a developer to code and integrate on top of those great Google and Applications APIs, or use tools like marvin. to help you automate some of the processes that are required to build a data warehouse.

marvin. is a tool from Bimotics that allows you to set up an online data warehouse without the need to code. Google has created the capability to host the biggest data warehouse in history, Google BigQuery, and marvin. is the bridge you need to make that process as easy as possible.  The steps below will help you get started.

What does it take to set up a Data Warehouse on the cloud? A simplified process for cloud bi would contain the following steps: 

1. Gathering the Data

2. Storing the Data

3. Analyzing the Data

4. Visualizing the Data

Gathering the Data

This first step has historically been the hardest, but advances in on-premise technology and Cloud applications like QuickBooks and SalesForce now provide methods to extract the data through APIs. These are often referred to as connectors. Most of these apps allow users to extract data in formats like CSV or JSON. Even e-commerce platforms allow users to extract their customer, product, order, inventory, and lead information through these connectors. Extracting these files to build a Cloud Data Warehouse is where it all begins. Without access to raw data, it is hard to design and build the right Data Warehouse architecture. Remember the old mantra “garbage in, garbage out”, bad data will always yield bad insight or analysis. Important note before going any further: if your data has exceptions, missing items, or other errors, fix them in the application itself before progressing with your warehouse.

Storing the Data

Storage is another big challenge.  You will need to answer questions such as:

• Where should I store this data and how?
• Is there enough space on my servers and hard drives?
• How much data needs to be stored?
• How often does information need to be added?

After setting marvin. up, pull data from your chosen application (e.g. e-commerce sample). marvin. then allows you to create buckets or folders where these files are to be stored- each relating to specific customers, products, orders, inventory etc.

Organizing data into buckets makes it much easier to refer to later. We suggest you create two types of buckets for each file: one to process into the next step and another for files that have already been processed. For example, we have a file called “Orders_New” and one called “Orders_History.” Not to go into great detail on file names, but naming them as order_YYYYMMDD.csv can help identify when the orders where extracted.

marvin. allows you to store data in any format, but the next step requires that the data be either  in CSV or JSON formats. marvin. will take an additional helpful step in compressing the files into GZIP. This minimizes the storage space used as well as the cost associated with storing the data. Storing your data this way allows you to keep a good archive of your data. marvin. also lets you to upload multiple files or folders into a bucket. Finally, you can preview files, as well as download or delete the ones that you do not need.

In the next part of this series, we will go into the next two steps: analyzing and visualizing data using marvin. 

Google BigQuery is a Big Deal : A Brief History of Database Evolution

As a person so close to, active in, and passionate about the technology industry, I love sharing my knowledge with non-technical crowds from time to time.  It is rewarding and refreshing to simplify some difficult concepts, take a look back at history and gain new perspective.  I recently found myself educating the less-technical side of our team at Bimotics about the incredible power of analytical databases like Google BigQuery. What I realized is that folks can better appreciate BigQuery’s  capabilities by  first understanding how databases have evolved over time.  The following is a short evolutionary history of databases and a summary of why BigQuery really is a big deal!

Writing

Most of the databases since the 1970s have been developed, designed and architected with the main purpose of writing and capturing data. Most of these "writing" databases are based on storing data in rows, as this makes it very simple to gather, append and collect data as quickly as possible. SQL became a very common language to inquire or query the data from these writing-optimized relational database management systems. As the volume of data has grown exponentially over the last 40 years, these writing-optimized databases have started to show their age in terms of analyzing large numbers of records, making analytics on top of these architectures time consuming and slow. Big data has also been focused on collecting everything, further increasing the size of and strain on these write databases.

Reading

Although the challenge of creating analytical databases or databases optimized for “reading” also started in the 1970s, it was not until the 1990s that reading-optimized database adoption  began to grow and mature. Multiple approaches like Cubes and non-SQL languages like MDX that focused on being able to read row based (“write”) databases more quickly, usually required processing or ingestion that took time and introduced latency. These approaches also made it hard for analysts to leverage “read” databases like HP’s Vertica, Cognos, Hyperion or Microsoft. Eventually, “reading” databases like SybaseIQ (1995) were developed and based on storing data in columns rather than  rows. This design optimized the capabilities to query and read information quickly. The next step was to move these analytical databases to the cloud.

Why Google BigQuery is a big deal

It was not until the mid-2000s that cloud databases based on columnar storage with SQL language compatibility became a real solution for analysts. Google BigQuery made it to market by 2010, allowing for unparalleled speed regardless of size. Based on a Dremel whitepaper from 2006, the reading optimized database has unlimited growth and unparallel performance. BigQuery is a big big deal in terms of analytical databases because it allows analysts to query their data very quickly using SQL regardless of the size. You can ingest data very fast, while never stopping to query thus removing the latency introduced by early analytical databases. BigQuery is available as a cloud service that does not require configuration or hardware setup.  It is auto optimized with no indexes to manage. Finally, it is fully secure and capable of complex nested structures to support web data. BigQuery is indeed a big deal and the product of 40 years of database evolution!

The next natural progression for Healthcare analytics

In building out our services practice, we have made a lot of headway consulting for small and medium sized businesses in the healthcare industry. With the 2015 government mandate to put electronic medical records in the cloud and make patient information more accessible to the patients themselves, we at Bimotics have seen healthcare providers exploring cloud based analytics options. Now that these providers have found ways to store records in the cloud, they are looking for cost effective methods to leverage this data for reporting and analysis.

With data already in the cloud, healthcare providers are seeing that capital costs needed for more administrative services can be minimized as spending for technology infrastructure such as hardware and structure can be left to the those providing cloud services. Cloud technology also benefits the patients with greater access to their information--even from different specialists and hospitals. With these increased efficiencies and accessibilities, the tangible evidence of cloud’s benefits is speeding its adoption in the healthcare industry.  

Adoption for healthcare based cloud business intelligence and analytics is not without its challenges and obstacles, however. HIPPA compliance, for example,  requires stringent privacy rules and has very serious repercussions when requirements are compromised. There is also much concern surrounding security and data integrity. Solutions can be engineered to address these concerns and we will go into the details of these technical solutions in upcoming blog installments.

Cloud-based healthcare analytics is definitely coming, and coming sooner than some may think.  Given the mandate to place patient records in the cloud and the associated advances enabling holistic patient care, healthcare providers and patients alike will demand equally cost-effective and accessible reports of the services they give and receive.

Related Blogs

10 Signs Your Business is Ready for Analytics Software

Data analysis findings in insurance fraud