Data Persistence

In this blog I would like to talk about data persistence..







•Information systems process data and convert them into information

•The data should persist for later use
•To maintain the status
•For logging purposes
•To further process and derive knowledge (data science)
•Data can be stored, read, updated/modified, and deleted
•At run time of software systems, data is stored in main memory, which is volatile
•Data should be stored in non-volatile storage for persistence

 Data
    Data is a set of values of subjects with respect to qualitative or quantitative variables. Data and information or knowledge are often used interchangeably; however data becomes information when it is viewed in context or in post-analysis.

  Database
   
A database is an organized collection of data, generally stored and accessed electronically from a computer system. Where databases are more complex they are often developed using formal design and modeling techniques.

   Database server
A database server is a server which houses a database application that provides database services to other computer programs or to computers, as defined by the client–server model.


  Database management system
A database management system (DBMS) is system software for creating and managing databases. The DBMS provides users and programmers with a systematic way to create, retrieve, update and manage data.A DBMS makes it possible for end users to create, read, update and delete data in a database.

Data persistence techniques 

•Data can be stored in
•Files
•Databases


Pros & cons of file system and database


Pros: File System

Performance can be better than when you do it in a database
Saving the files and downloading them in the file system is much simpler
Migrating the data is an easy process.

It's easy to migrate it to cloud storage

Cons:

Loosely packed
Low security


Pros:  Database

    ACID consistency, which includes a rollback of an update that is complicated when files are stored outside the database.
    Files will be in sync with the database and cannot be orphaned, which gives you the upper hand in tracking transactions.
    Backups automatically include file binaries.
    It's more secure than saving in a file system.


Cons of Database

    You may have to convert the files to blob in order to store them in the database.
    Database backups will be more hefty and heavy.
    Memory is ineffective.



 Data arrangement                          
•Un-structured
    Examples include e-mail messages, word processing documents, videos, photos, audio files, presentations, webpages and many other kinds of business documents
•Semi-structured     CSV but XML and JSON documents are semi structured documents

•Structured                                                             
  numbers, dates, and groups of words and numbers called strings

   Database type 

•Hierarchical databases
•Network databases
•Relational databases
•Non-relational databases (NoSQL)
•Object-oriented databases
•Graph databases
•Document databases


Data warehouse and Bid data

data warehouse
  In computing, a data warehouse, also known as an enterprise data warehouse, is a system used for reporting and data analysis, and is considered a core component of business intelligence. DWs are central repositories of integrated data from one or more disparate sources.

Big data
"Big data" is a field that treats ways to analyze, systematically extract information from, or otherwise deal with data sets that are too large or complex to be dealt with by traditional data-processing application software.


Both of them hold a lot of data, used for reporting, managed by an electronic storage device. So one common thought of maximum people that recent big data will replace old data warehousing very soon. But still, big data and data warehousing is not interchangeable as they used totally for a different purpose. So let us start learning Big Data and Data Warehouse in a detail in this post.

Files and DBs are external components
•They are existing outside the software system
•Software can connect to the files/DBs to perform CRUD operations on data
•File –File path, URL
•DB –connection string

Prepared statements

•The query only needs to be parsed (or prepared) once, but can be executed multiple times with the same or different parameters.

PreparedStatementpstmt= con.prepareStatement("update STUDENT set NAME = ? where ID = ?");
pstmt.setString(1, "MyName"); pstmt.setInt(2, 111); pstmt.executeUpdate();

Callable statements •Execute stored procedures
CallableStatementcstmt= con.prepareCall("{call anyProcedure(?, ?, ?)}"); cstmt.execute();



 Object-relational mapping
Object-relational mapping (ORM, O/RM, and O/R mapping tool) in computer science is a programming technique for converting data between incompatible type systems using object-oriented programming languages. This creates, in effect, a "virtual object database" that can be used from within the programming language.

There are different structures for holding data at runtime •Application holds data in objects •Database uses tables (entities)

Mismatches between relational and object models
•Granularity
•Subtypes
•Identity
•Associations
•Data navigation

ORM implementations in JAVA
•Java Beans
•JPA

Beans use POJO stands for Plain Old Java Object. It is an ordinary Java object, not bound by any special restriction other than those forced by the Java Language Specification and not requiring any class path. POJOs are used for increasing the readability and re-usability of a program.

A POJO should not:
•Extend pre-specified classes.
•Implement pre-specified interfaces.
•Contain pre-specified annotations.

Beans
•Beans are special type of Pojos. There are some restrictions on POJO to be a bean.
•All JavaBeans are POJOs but not all POJOs are JavaBeans.
 •Serializable i.e. they should implement Serializable interface. Still some POJOs who don’t implement Serializable interface are called POJOs because Serializable is a marker interface and therefore not of much burden.

Java Persistence API (JPA)    
•Uses
•POJO classes
•XML based mapping file (represent the DB)
•A provider (implementation of JPA)

NOSQL AND HADOOP
Not Only SQL (NOSQL)

•Relational DBs are good for structured data
•For semi-structured and un-structured data, some other types of DBs can be used
•Key-value stores
•Document databases
•Wide-column stores •Graph stores


Benefits of NoSQL
 •When compared to relational databases, NoSQL databases aremore scalable and provide superior performance,and their data model addresses several issues that the relational model is not designed to address:
•Large volumes of rapidly changing structured, semi-structured, and unstructured data

NoSQL DB servers
•MongoDB
•Cassandra
 •Redis
•Amazon DynamoDB
 •Hbase

Information retrieval (IR) is the activity of obtaining information system resources relevant to an information need from a collection. Searches can be based on full-text or other content-based indexing. Information retrieval is the science of searching for information in a document, searching for documents themselves, and also searching for metadata that describe data, and for databases of texts, images or sounds.

•The information retrieval process should be
 •Fast/performance
•Scalable
•Efficient
•Reliable/Correct

Read More

Client-side development 2 - RiWAs

In this blog, I would like to give a  Introduction to the RiWAs in THE CLIENT-SIDE DEVELOPMENT  that we see today and how they are being utilized in our day to day applications.






Rich internet application (RIA)


                 A rich internet application (RIA) is a web application designed to provide the same functions and features that are normally associated with desktop applications. By locating the user interface and related activity and capability on the client side, and by manipulating and operating data on the application server side, RIAs generally divide processing across the Internet / network divide.

An RIA normally runs within a web browser and does not usually require client-side software installation to work. However, with one or more specific browsers, some RIAs can only work properly. Within a special isolated area of the client desktop called a sandbox, most RIAs run their client portions for security purposes. The sandbox limits on the other side of the connection visibility and access to the client's file and operating system to the application server.

 Rich web application (RiWA)
 
                       A rich web application (originally called a Rich Internet Application RIA or Installable Internet Application) is a web application with many desktop application software features, typically delivered via a site-specific browser, a browser plug-in, an independent sandbox, extensive use of JavaScript, or a virtual machine.

The concept is closely related to a single-page application and can provide the user with interactive features such as drag and drop, background menu, editing WYSIWYG, etc. HTML5 is a current standard, supported by all major browsers, for delivering rich web applications.

 Key features of RiWAs

Direct interaction:

 In a traditional page-based Web application, interaction is limited to a small group of standard controls: checkboxes, radio buttons and form fields.
This severely hampers the development of applications that are usable and engaging. An RIA can use a wider range of controls to make the user experience more efficient and better.

Partial-page updating:

Standard HTML-based Web pages are loaded once. If you update something on a page, the change must be sent back to the server, which makes the changes and then resends the entire page. Standard HTML-based Web pages are loaded once. If you update something on a page, the change must be sent back to the server, which makes the changes and then resends the entire page.
With HTTP and HTML, there is no other way to do it. With traditional web-based apps, problems with network connectivity, limitations in processing and other issues require users to wait while the entire page reloads.

Offline use:

When connectivity is unavailable, it might still be possible to use an RIA if the app is designed to retain its state locally on the client machine. (Developments in Web standards have also made it possible for some traditional Web applications to do that.)


 Client Side Scripting / Coding - Client Side Scripting is the type of code that is executed or interpreted by browsers.

 Client Side Scripting is generally viewable by any visitor to a site (from the view menu click on "View Source" to view the source code).
Below are some common Client Side Scripting technologies:

    HTML (HyperText Markup Language)
    CSS (Cascading Style Sheets)
    JavaScript
    Ajax (Asynchronous JavaScript and XML)
    jQuery (JavaScript Framework Library - commonly used in Ajax development)
    MooTools (JavaScript Framework Library - commonly used in Ajax development)
    Dojo Toolkit (JavaScript Framework Library - commonly used in Ajax development)

  Delta Communication Technologies

Delta-Communication is the rich communication model used by the RIA's rich features, for the client-component(s) to communicate with the server-component(s), to exchange only the necessary data set – for a particular feature executed at the time – which is smaller than the size of the traditional communication request / response.
Since the size of the communicated data set is smaller, the communication is completed more quickly, eliminating the pattern of work-wait. 





XHR/AJAX

Ajax is a set of web development techniques using many web technologies on the client side to create asynchronous web applications. With Ajax, web applications can send and retrieve data from a server asynchronously (in the background) without interfering with the display and behavior of the existing page. By decoupling the data interchange layer from the presentation layer, Ajax allows web pages and, by extension, web applications, to change content dynamically without the need to reload the entire page.In practice, modern implementations commonly utilize JSON instead of XML.


XMLHttpRequest (XHR) is an API in the form of an object whose methods transfer data between a web browser and a web server. The object is provided by the browser's JavaScript environment. Particularly, retrieval of data from XHR for the purpose of continually modifying a loaded web page is the underlying concept of Ajax design. Despite the name, XHR can be used with protocols other than HTTP and data can be in the form of not only XML,but also JSON, HTML or plain text.

WHATWG maintains an XHR standard as a living document. Ongoing work at the W3C to create a stable specification is based on snapshots of the WHATWG standard.  

 Ajax

The jQuery library includes various methods to send Ajax requests. These methods internally use XMLHttpRequest object of JavaScript. The following table lists all the Ajax methods of jQuery.

jQuery Ajax Methods	Description
ajax()	Sends asynchronous http request to the server.
get()	Sends http GET request to load the data from the server.
Post()	Sends http POST request to submit or load the data to the serve
getJSON()	Sends http GET request to load JSON encoded data from the server.
getScript()	Sends http GET request to load the JavaScript file from the server and then executes it.
load()	Sends http request to load the html or text content from the server and add them to DOM element(s).

 

jQuery ajax() Method

The jQuery ajax() method provides core functionality of Ajax in jQuery. It sends asynchronous HTTP requests to the server.

Syntax:

           $.ajax(url);

           $.ajax(url,[options]);

           Parameter description:

    url: A string URL to which you want to submit or retrieve the data
    options: Configuration options for Ajax request. An options parameter can be specified using JSON format. This parameter is optional.

 

 

Options	Description
accepts	The content type sent in the request header that tells the server what kind of response it will accept in return.
async	By default, all requests are sent asynchronously. Set it false to make it synchronous.
beforeSend	A callback function to be executed before Ajax request is sent.
cache	A boolean indicating browser cache. Default is true. complete      A callback function to be executed when request finishes.
contentType	A string containing a type of content when sending MIME content to the server.Default is "application/x-www-form-urlencoded; charset=UTF-8"
crossDomain	A boolean value indicating whether a request is a cross-domain.
data	A data to be sent to the server. It can be JSON object, string or array.
dataType	The type of data that you're expecting back from the server.
password	A password to be used with XMLHttpRequest in response to an HTTP access authentication request.

                                        Category: Shorthand Methods

 

Read More