17.07M

Data and Business 1

Volume of DATA in the world
175
(in zettabytes)
130
Big Data is complex and voluminous sets of different Data.
They are presented in raw form and require pre-processing. To obtain valuable
information from them that can benefit businesses and organizations, you need
to use various tools, approaches and methods for their processing
2
5
2010 2011
7
2012
26
18
16
13
9
2013 2014
2015
2016
2017
41
33
2018
2019
51
65
80
2025
101 2024
2023
2022
2021
2020
175
Zettabytes of data
by 2025

3.

Data

4.

Data
information, facts characterizing someone, something,
necessary for any conclusions, decisions
known facts used for inference or calculation

5.

6.

7.

Artificial
Intelligence
Data
Analytics

8.

Artificial
Intelligence
Data
Analytics

9.

INFORMATION

10.

Information
information about the surrounding world and the
processes occurring in it, perceived by humans or
special devices
something that has been communicated; knowledge
it is the removed uncertainty of our knowledge about
something
(Claude Elwood Shannon)

11.

12.

13.

Information
Information is subjective and
depends on the purpose of its
consumption

14.

Wisdom
Knowledge
I know how
Information
I know that
Data
I do not know anything
I know why

15.

Data
It's raining

16.

Information
The temperature dropped 15
degrees and then it started to
rain

17.

Knowledge
If the humidity is high and the
temperature drops, the moisture
is unlikely to be able to stay in
the atmosphere, so it usually
rains

18.

Wisdom
It's raining because it's raining.
This includes understanding all
natural interactions

19.

Wisdom
Knowledge
INFORMATION
DATA

20.

Corporate Data
Structured Data
Unstructured Data

21.

Structured Data
Identifier*
Name
Surname
Phone
1
Ivan
Ivanov
555-845-21
2
Peter
Petrov
555-415-14
3
Ian
Latimer
555-130-46
4
Ann
Loonberg
555-991-01
Relation
Attributes
Attribute
“Surname”
Key
Petrov
Value
“Person”
Relation
“2”
Key

22.

Unstructured Data
located
Munich
like
Germany
Ann
person
car
Ivan
person
Irina
friends
built
Root
person
married
drive
BMW
work with
Peter
work in
Andersen
person

23.

“Data is the new oil”
Clive Humby
15%
Business
relevant
data
business
potentially
relevant
data
Business
irrelevant
data
50%
Repeatable
Data
35%
?
50%
Non-Repeatable
Data
50%

24.

Expert
systems
Wisdom
Knowledge
OLAP
Systems
Support systems
decision making
INFORMATION
DATA
OLTP
Systems

25.

Data Algorithms
ML, AI
Data Analytics
Data Mining
Wisdom
Knowledge
INFORMATION
Data Transformation
SQL
DATA

26.

DATA
INFORMATION
Knowledge
Wisdom

27.

S* - scalable component
HA* - high available component
Notification Center
sms / messenger / email / push
Raw Layer
• customers
• customer service
• customs
• QA engineers
Operational Layer
Stage Layer
ready pack of documents
+
Declaration
by email
by REST API
Initial Photo
+
React JS
Frontend
Processed Photo
Declaration
RawToBroker
S*
HA*
Business
Process
Flow
• Block Data
• Quality of Recognition
• Metadata
PostgreSQL
OLTP
Apache
Kafka
HA+S*
DWH integration
Internal Service
Block / text
Recognition
HA+S*
BrokerToStage
• User feedback
• QM findings
• customs authorities response
• Business Process Flow
47%
Centralised
OLAP DB
Confidence Rating Metric
Reporting
System
Confidence Rating Report
Continuous Improvement of Confidence Rating
Internal Service
Processing Layer
Neural Network
Topology
HA*
REST API
S*
Continuous Learning of Neural Network
Spring + Spring Boot
Frameworks
Java Backend
Business
Goal

28.

S* - scalable component
HA* - high available component
Notification Center
sms / messenger / email / push
Raw Layer
• customers
• customer service
• customs
• QA engineers
Operational Layer
Stage Layer
ready pack of documents
+
Declaration
by email
by REST API
Initial Photo
+
React JS
Frontend
Processed Photo
Declaration
RawToBroker
S*
HA*
Business
Process
Flow
• Block Data
• Quality of Recognition
• Metadata
PostgreSQL
OLTP
DWH integration
• User feedback
• QM findings
• customs authorities response
• Business Process Flow
47%
HA+S*
Block / text
Recognition
HA+S*
BrokerToStage
Centralised
OLAP DB
Confidence Rating Metric
Reporting
System
Confidence Rating Report
S*
Neural Network
Topology
Continuous Improvement of Confidence Rating
Internal Service
Processing Layer
Continuous Learning of Neural Network
HA*
REST API
Internal Service
Apache
Kafka
Spring + Spring Boot
Frameworks
Java Backend

29.

Программа РЦТ ООВО

30.

31.

DATA
= RESOURCE

32.

DATA properties
Immateriality
Possibility of sharing
Copy-ability
Transportability
Versatility
Renewability

33.

DATA
= ASSET

34.

Data Management Challenges
Data fragmentation
in organization
impossibility
prompt receipt
access to organization data
high cost of IT
solutions
it is not clear how
to use data for the
benefit of the
organization
information is
not an asset of
the organization

35.

36.

37.

38.

39.

40.

41.

42.

Data Engineer
DataOps Engineer
Data Science Engineer
MLOps Engineer
Data Quality Engineer
BI Engineer
Data Architect
ETL Engineer
Data Analyst
PROMPT Engineer

43.

Deep Learning engineer
Data Analyst
ML architect
MLOps engineer
Data Science engineer
Data Analyst
ML architect
MLOps engineer
Data engineer
Data analyst
Data Architect
ETL engineer
Data engineer
Data analyst
Data architect
DataOps engineer
Data Quality engineer
Data journalist
Wisdom
Knowledge
INFORMATION
DATA

44.

Data Platform ROADMAP
For PASHA Insurance

45.

46.

47.

0
define the 1st Business Goals (final Data Products)
0

48.

1
to choose a database for the Data Warehouse
1

49.

describe integration protocols with the sources
2
(to cover the described data products)
2

50.

3
describe transport layer including CDC and Batch
(to cover the described data products)
3
3

51.

4
to make a technical pipelines to gather data in the DWH
(to cover the described data products)
4
4
4

52.

5
to integrate transformation layer
(to cover the described data products)
5
5

53.

6
to implement of the described Data Products
6

54.

7
to integrate a DataLake
7

55.

8
to make an integration between DWH and DataLake
(for instance, to cover Data Life Cycle)
8

56.

9
to make an integration CDC / Batch processes
9

57.

1
0
to support real time analytics
1
0
1
0
1
0

58.

1
1
to support stream processing for Scoreboards

59.

Roadmap Milestones
0
PostgreSQL DB
Python
Apache Airflow
DBT modelling
Apache Kafka
PowerBI
to define priority list of the
Business Goals (reports, dashboards)
Ordinary FileStorage
Apache Superset
1.1
to define Data Catalog and
Data Lineage models
1.2
to define MDM / RDM
models
1
to choose PostgreSQL database
(the first version of the DWH)
2
to define needed data sources and
describe data protocols
3
make integrations using CDC / batch patterns
with data sources
4
make integrations with DWH
5
integrate data transformation layer
Hadoop ecosystem (HDFS)
GreenPlum DB
Stream processing
ML development
Data Mining algorithms
11
To change
DataLake into
HDFS cluster
10
9
6
7
8
integrate Reporting layer
DataLake
LakeHouse
To change
DWH into
GreenPlum
DB

60.

Data Platform ROADMAP
for Limango company

61.

Business Goal
Continuous customer traffic to optimize customer value and overall business goals
Data Engineering
Area
create data pipelines
monitoring
support data pipelines
create / support ETL
Data Quality checks
Data Restoration
Data Science
Area
Data Analytics
Area
create ML pipelines
confirm hypothesis
collaborate with DA
continuous learning ML
Data Mining stuff
take a Value from data
analyse data
make experiments
make hypothesis
research user events
create initial MVP
find Insights
make A/B tests

62.

Proposal #1 for Data Engineering Area (improve batch data)
Client Data Center
1
ODS Layer (~ Stage Layer)
is mainly Type #2 (availability of business data during hours)
but for some tables Type #3 (availability of business data during 1 day)
…
1
Current situation is Batch Jobs
Client Data Center
2
Parallel ETL processes (job-for-table)
3
Less granularity time schedules
ODS Layer is Type #2 (minutes - hours)
…
3
4
2
…
4
…
…
More ETL branches to improve parallelism

63.

Proposal #2 for Data Engineering Area (real time data)
Client Data Center
1
…
1
Current situation is Batch Jobs
ODS Layer (~ Stage Layer)
is mainly Type #2 (availability of business data during hours)
but for some tables Type #3 (availability of business data during 1 day)
Client Data Center
2
Current flow to achieve a backward compatibility
3
MySQL Database Log (built in component in MySQL)
4
Tool reads database events and send to connector
5
Transport Layer to achieve continuous data
2
…
3
binlog
…
translator
Debezium
4
5

64.

Proposal #2 for Data Engineering Area (real time data)
Client Data Center
Current situation is Batch Jobs
1
…
1
ODS Layer (~ Stage Layer)
is mainly Type #2 (availability of business data during hours)
but for some tables Type #3 (availability of business data during 1 day)
2
Current flow to achieve a backward compatibility
3
MySQL Database Log (built in component in MySQL)
4
Tool reads database events and send to connector (CDC)
5
Transport Layer to achieve continuous data
6
Worker with translation events to SQS queue
Client Data Center
2
…
3
binlog
…
translator
Lambda
Funcations
Debezium
6
4
This design provides possibility to create ODS Type #1
5
S3 storage

65.

Proposal #3 for Data Engineering Area (real time data)
Client Data Center
1
…
1
Current situation is Batch Jobs
ODS Layer (~ Stage Layer)
is mainly Type #2 (availability of business data during hours)
but for some tables Type #3 (availability of business data during 1 day)
2
AWS Amazon Kinesis Data Streams
Client Data Center
…
binlog
Lambda
Funcations
translator
S3 storage
Amazon EMR
Debezium
2
This design provides possibility to create ODS Type #1
Kinesis
Data Analytics
Output

66.

67.

Proposal for Data Science / Analytics / MDM Areas
Amazon EMR
MDM Layer
ODD / ODS Layer
Raw Layer
AWS Athena
REST API
• Store hyper parameters for NN / ML per user
• Store recommendations per user
• Store user groups
• Store similarities between users
• etc.
Mail Server
Automated Mailing of Item Catalogues
and personally sales based on
recommendations and history
Researchs
Insights
Experiments
Real Time Calculations
Lieber Kunde,
Wir möchten Ihnen und für Ihre Kinder
unsere Neuheiten zeigen ...
Lieber Kunde,
Wir danken Ihnen für Ihr Interesse an
unserem Shop und erstellen Ihnen gerne
ein persönliches Angebot ...