Computer
Computer Diary
    RepRap


    "God is not external to anyone, but is present with all things, though they are ignorant that he is so."

    Plotinus
    204-270 AD

    Here my little rant and praise place, where the daily experiences of my programming work are expressed. I publish them with the idea that others might find it useful and benefit from it.

    2013/12/14
    MetaFS - Dealing With Metadata the Proper Way
    last edited 2013/12/20 16:04 (*)

    As I posted earlier Metadata - The Unresolved Mess (2012-07) I have started another effort to really bring decade long pondering on the issue together, and MetaFS I started a few weeks ago, and it is slowly growing now:


    MetaFS is a "Proof of Concept" meta file system which:

    • has no hierarchical constraints, but can have a hierarchical appearance
    • free definable metadata & tags for each item
    • full text indexing & search on the fly
    • content is hashed by default (e.g. maintaining integrity & finding duplicates)

    It has been implemented for Linux using FUSE, specifically using the Perl-FUSE module, providing UNIX file system interface for test purposes:

    % cp .../Shakespeare/*.txt .
    

    % cp .../Books/*.pdf .

    % cp -r .../Photos .

    once data is in the MetaFS realm, all data is indexed within seconds (it may take longer if large quantity of new data arrives), as next you can query the indexed data:
    % mfind sherlock
    ... list all files where 'sherlock' was found (.txt, .pdf, .odt etc)
    

    % mfind mime:image/ ... list all images

    % mfind location:lat=40,long=-10 ... list all items (e.g. photos) with near GPS location

    % mfind location:city=London,GB ... list all items (e.g. photos) with near GPS location

    "No more grep", but all content is fully indexed.

    I think a state-of-the-art file system should have the functionality of a database:

    • easy to query
    • fast searching of content, filename, tags or other metadata (incl. geospatial lookup of GPS coordinates of items)
    • flexible views (no constraints on actual structure)

    Linux desktops (GNOME, KDE, etc) and Microsoft's Windows (XP,Vista,7,8) have all failed (as of 2013/11) in this regards, metadata information are not handled well, search for content are only available as third party software or are very slow (e.g. post indexing). Apple on the other hand has provided some of the functionality as MetaFS proposes: OS X provides full text search and tagging; yet, it's only available on the OS X platform and is a closed system.

    MetaFS goes further, beside being Open Source, it allows you to write your own handlers, e.g. to extract metadata from soundfiles and visualize the waveform, or parse the text content and look for village or city names, and tag the text with GPS coordinates.

    You find more information at MetaFS.org .



    All posts or individual posts:

  • MetaFS - Dealing With Metadata the Proper Way (2013/12/14 01:00)
  • My Cellphones & Smartphone (2010-2013) (2013/09/02 23:15)
  • KDE / Kubuntu 12.04: 10+ years terrible GUI, A Systemic Problem of OSS (2013/04/27 12:18)
  • UNIX Man on Windows 7: VirtualBox + Ubuntu + LXC (2012/11/27 19:24)
  • Metadata - The Unresolved Mess (2012/07/10 20:59)
  • Cellphone Networks: Thieves, Insanity & Crap (2010/01/26 13:57)
  • MacOS-X for a UNIX Man with a PC (2009/09/26 20:43)
  • Windows XP for a UNIX Man (2009/09/22 18:28)
  • Server Counting (2009/05/18 11:50)
  • Automatically Geotag Photos without GPS (2009/04/22 08:28)
  • Rebirth of FastCGI (2009/04/15 17:17)
  • Online Advertisement & Income for Web-Site Owners (2009/03/18 22:10)
  • iPhone JavaScript Frameworks (aka Avoiding Objective-C) (2009/03/14 22:08)
  • Google - The Almighty Tracker & Advertising Blocking (2009/03/12 22:09)
  • How To Save 300MB RAM (2009/03/07 22:07)
  • Verbosity of Programming Languages (2009/03/06 22:06)
  • Problems with MacOSX (2009/03/03 22:03)
  • MacOSX: My First Steps (2009/02/24 09:57)
  • Catch 22 with HDD/DVD Recorder Medion Life (2009/02/24 09:27)
  • Kubuntu 8.1 as guest on VirtualBox MacOSX host (2009/02/24 01:33)
  • VirtualBox vs VMWare Fusion on MacOSX (2009/02/24 01:19)
  • SQL vs GREP with 230K lines (12MB) GeoLite (2009/02/23 20:10)
  • Kubuntu 8.1: Eye-Candy & Memory Waste (2009/01/24 09:57)
  • Firefox 2.0.x / 3.0.x - Memory Waste (2009/01/22 19:34)


    [ post new entry ] (only for administators)

    Title:

    Text:

    Tags: (separated by commas)

    Date (optional):

    Password:
  •  


    .:.




    Copyright 2007-2016, 2020-2024 © by René K. Müller <spiritdude@gmail.com>
    Illustrations and graphics made with Inkscape, GIMP and Tgif