From Knelcorpwiki

Jump to: navigation, search

Data Ownership Project

This wiki page is where I'm capturing all things related to data ownership.

Delivered presentations

I've presented on this a few times.

SFVLUG - March 6th 2010 - download pdf
UUASC - March 8th 2010 - download pdf
SGVLUG - March 11th 2010 - no download. same as others, just different venue/date.

Why am I doing this?

Hosted services fail in spectacular ways on a way to often basis.

I help my clients build systems and processes/policies to own their data. I need to drink my own champagne
It sounds cool?

How am I going about it?

Step 1: Map out where my existing data is

Data that is generated by daily activities
- health data (stored on servers at my health care provider. Generally not easily accessible)
- financial data (stored by financial institutions. Generally easily accessible in the form of CSV dumps)
- search data (stored by search engines. Generally not accessible)
- requests for directions (stored by various providers. Generally not accessible)

Data that is created by me
- Textual
  - Blogs (often hosted on wordpress/blogspot. easily accessible and movable)
  - Tweets (hosted on twitter. easily accessible)
  - Social networking posts (hosted on various sites. accessibility varies)
- audio
  - Podcasts
  - Voice notes
- video
- pictorial

Step 2: Evaluate replacement systems

Type of service/application	What I was using before and/or what others are commonly using	What I have migrated to	Notes/URL
News	Google news/google reader	Dashboard ,Tattler and rss2email	http://dashboard.knownelement.com (haven't deployed tattler yet). In the very early stages of using this system. Existing usage I have seen is quite cool.
Photos	flickr	gallery2	http://photos.knownelement.com (imported all my flickr photos and now upload straight here)
Microblogging	Twitter	status.net	http://mblog.knownelement.com (bridges my posts to twitter, pulling posts in from twitter fails a lot due to twitter scalability issues)
Blogging	livejournal/blogspot	wordpress	http://blog.knownelement.com (very happy with wordpress. imported all my lj posts, need to pull in my blogspot posts)
Issue tracking and software project management	Basecamp/lighthouse/sourceforge/google code	redmine	http://redmine.knownelement.com:3000 (was a heavy trac user, but redmine won me over with the modern UI, thick client and sub tasks/tickets)
Invoicing clients	Bye bye freshbooks / simple text files / spreadsheets generated manually	argentuminvoice	http://invoices.knownelement.com (pretty happy with this software. produces very nice looking invoices)
URL shortener	Bye bye tr.im/tinyurl	casimir	http://url.knownelement.com
Knowledge management	(never really used any other systems other then text based note files)	mediawiki	http://wiki.knownelement.com
Centralized login	claimid/facebook connect	Active Directory/RADIUS and OpenID (phpMyID)
Collaboration	Google docs/skype/webex/instant messaging	Openfire jabber server with Kraken for aim/msn/yahoo/irc interop and Karaka for skype interop, askterisk/freeswitch/pbxinaflash for secure voice chat, openmeetings/ bigbluebutton for webex replacement and eyeos for collaborative document editing.	http://desktop.knownelement.com hosts my eyeos instance, jabber is only available via the vpn. Voice stuff and Bigbluebutton/openmeetings not yet deployed. That's really it's own project.
Data Sync (between mobile devices and "the cloud"	Google/yahoo sync software, active sync, bes	funambol + microsoft exchange
CRM	salesforce.com	missing application
E-mail	Gmail	Missing application	N/A
Note taking	evernote	missingapp
fleet tracking	latitude	missing application (evaluating http://opengts.org/)
Calendar/tasks	Thunderbird+Sunbird/Outlook	missing application

Step 3: Assemble the infrastructure for hosting

Physical infrastructure

Servers

1 rack mount system running a Ubuntu VM (primary)
1 dell optiplex system (hot backup)

Details of configuration can be found at Main-server_documentation wiki page and Media_center_setup wiki page (I use my HTPC as my secondary system).

Network Gear

1 Motorola DSL modem
1 Cisco 1841 router
1 Cisco 3548 switch

Details of configuration can be found at Network_Stuff#Production_Network wiki page.

Other Gear

APC UPS (cisco gear plugged into surge only, usb drive and dell optiplex plugged into battery backup)
1TB USB drive

Server Software

E-mail bits

Sending e-mail from a "dial up" IP range can be a royal pain. Some things to help:

Found

http://dialup.pacbell.net/dn_worksheet.html << this one is key. gets you off a lot of black lists if you control both forward/reverse dns. :)

via http://www.dslreports.com/faq/14282



For DSL DNS Needs, call:
DSL Provisioning 800-833-2120 Options 1, 2, 1

For Dedicated Access DNS Needs, call:
Dedicated Enhanced Service Center (DESC) 1-866-937-3664, Options 3,5

For SBCIS Sales, call:
1-888-724-7253

For Web Hosting services, call:
Web Hosting Sales: 888-WEB-HOST (1-888-932-4678)

Jabber Bits

Openfire setup guide at http://library.linode.com/real-time-messaging/xmpp-servers/install-openfire-ubuntu-9.10-karmic

Web Software Bits

LDAP Software Bits

Took me a bit of searching. Came across

Also some tools:

http://www.jxplorer.org/

Client Software

E-mail

SIP

Jabber

Step 4: Migrate Data

Step 5: Host data in a sustainable fashion

Backups:

s3
- cost effective
- off site
- easy to implement
- you wanted to actually restore those backups? :)

local storage
- cost effective
- want off site and rotation? just buy a few drives
- easy to implement
- restores very nicely

replication
- mysql (instructions here)
- dns (instructions here)
- apache
- linux ha

Security:

Software/logical

snort/securita
greensql
logwatch
openvas
awstats

Physical

guns
alarm systems
dogs
bolting the gear to the rack :)

Monitoring:

Internal

OpsView
Netdisco
Rancid
apt-get install logwatch snort and set root to a live e-mail address. Provides an excellent daily summary of activity on the system.

External

Nothing at the moment.

Get all the details at my Monitoring_Alerting_and_Network_visualisation wiki page.

Data Ownership