lmdb

Does LMDB support multiple keys to same value mapping?

由 和自甴很熟 提交于 2021-01-27 16:33:42
问题 is it possible to have multiple keys mapping to the same value? If not, is there a work around for this feature? 回答1: It isn't possible. One workaround that I use is to have the value on the second key be a pointer to the primary key. That is, the value of the second key is the primary key. In particular, I make a secondary-keys table (or "Named Database" in lmdb speak) where all the values are primary keys in the primary table. If you look further into other database this is exactly how they

SSD目标检测lmdb数据结构剖析

由 强颜欢笑 提交于 2020-01-26 00:26:41
SSD读取训练集是从LMDB中读取AnnotatedDatum结构的数据，在训练和测试之前，要将图片（img）和XML（label）数据存储为AnnotatedDatum结构，然后将数据经过序列化，存入到LMDB数据库中。训练和测试的时候直接从LMDB读取数据，经过反序列化获取AnnotatedDatum结构的数据，获得训练集的图片和XML数据。 可以参考ssd caffe目录包下的src/caffe/util/io.cpp, tools/convert_annoset.cpp,会对你理解数据结构有很大的作用。 也是C++ 强大的动态内存管理推波助澜。 AnnotatedData数据结构 message AnnotatedDataParameter { // Define the sampler. repeated BatchSampler batch_sampler = 1; // Store label name and label id in LabelMap format. optional string label_map_file = 2; // If provided, it will replace the AnnotationType stored in each // AnnotatedDatum. optional AnnotatedDatum

Caffe训练（3）图像数据转换成db（leveldb/lmdb)文件

由 戏子无情 提交于 2020-01-23 01:27:36
caffe训练（3） convert_imageset.cpp参数说明 txt文件 FLAGS: 图片参数组 生成lmdb 例子 在深度学习的实际应用中，我们经常用到的原始数据是图片文件，如jpg,jpeg,png,tif等格式的，而且有可能图片的大小还不一致。而在 caffe中经常使用的数据类型是lmdb或leveldb ，因此就产生了这样的一个问题：如何从原始图片文件转换成caffe中能够运行的db（leveldb/lmdb)文件？ convert_imageset.cpp参数说明 caffe中，作者为我们提供了这样一个文件： convert_imageset.cpp ，存放在根目录下的tools文件夹下。编译之后，生成对应的可执行文件放在 buile/tools/ 下面，这个文件的作用就是用于将图片文件转换成caffe框架中能直接使用的db文件。 该文件的使用格式： convert_imageset [FLAGS] ROOTFOLDER/ LISTFILE DB_NAME 需要带四个参数： FLAGS: 图片参数组，后面详细介绍 ROOTFOLDER/: 图片存放的绝对路径，从linux系统根目录开始 LISTFILE: 图片文件列表清单，一般为一个txt文件，一行一张图片 DB_NAME: 最终生成的db文件存放目录 重点介绍 FLAGS、 LISTFILE两个参数。

Creating large LMDBs for Caffe with numpy arrays

由 邮差的信 提交于 2020-01-16 05:21:04
问题 I have two 60 x 80921 matrices, one filled with data, one with reference. I would like to store the values as key/value pairs in two different LMDBs, one for training (say I'll slice around the 60000 column mark) and one for testing. Here is my idea; does it work? X_train = X[:,:60000] Y_train = Y[:,:60000] X_test = X[:,60000:] Y_test = Y[:,60000:] X_train = X_train.astype(int) X_test = X_test.astype(int) Y_train = Y_train.astype(int) Y_test = Y_test.astype(int) map_size = X_train.nbytes * 10

How to read image from numpy array into PIL Image?

由 天大地大妈咪最大 提交于 2020-01-11 08:24:21
问题 I am trying to read an image from a numpy array using PIL, by doing the following: from PIL import Image import numpy as np #img is a np array with shape (3,256,256) Image.fromarray(img) and am getting the following error: File "...Image.py", line 2155, in fromarray raise TypeError("Cannot handle this data type") I think this is because fromarray expects the shape to be (height, width, num_channels) however the array I have is in the shape (num_channels, height, width) as it is stored in this

Forward from the first of lmdb when using net.forward in pycaffe

由 吃可爱长大的小学妹 提交于 2020-01-05 05:36:08
问题 I am using pycaffe and my train and test data is in LMDB format. I have created my net like this: net = caffe.Net('train.prototxt', 'c.caffemodel', caffe.TEST) when you call net.forward, implicitly you walk through the LMDB test database one by one batches. My question is how can I start from the beginning of LMDB and test my network on the first n batches of the test data? Thanks 回答1: not sure it still relevant but you would need to change the data of input layer ,something like this net

Windows下caffe安装详解(仅CPU)

由 狂风中的少年 提交于 2020-01-04 16:40:41
本文大多转载自 http://blog.csdn.net/guoyk1990/article/details/52909864，加入部分自己实战心得。 1、环境：windows 7\VS2013 2、caffe-windows准备 (1) 下载官方caffe-windows 并解压，将 .\windows\CommonSettings.props.example备份，并改名为CommonSettings.props。如图4所示： 图 4：修改后的CommonSettings.props文件 附带说明，现在最新版的github已经更新，没有上述文件，根据大佬说法用cmake编译后能产生sln文件，笔者不才，并不会，这里提供百度云盘的老版本： caffe提供Windows工具包(caffe-windows)：https://github.com/BVLC/caffe/tree/windows 百度云下载地址：链接：http://pan.baidu.com/s/1bp1BFH1 密码：phf3 (2)关于CommonSettings.props文件的一点说明。 [html] view plain copy </ pre > < pre name= "code" class= "html" > <? xml version= "1.0" encoding= "utf-8" ?> <

Count number of records in lmdb databse with python

由 大憨熊 提交于 2020-01-02 08:11:12
问题 I open a lmdb database using this code: lmdb_env = lmdb.open(source_path, readonly=True) How can I count the number of records in this database? 回答1: I think it should be like this: lmdb_env = lmdb.open(lmdb_file_name, readonly=True) print lmdb_env.stat() Then it prints the directory that Jaco pasted here. 回答2: You can use event.stat() . It will return the following dictionary with entries detailing the number of records in this database: {'branch_pages': 1040L, 'depth': 4L, 'entries':

Compressing LMDB files

由 会有一股神秘感。 提交于 2019-12-24 08:43:33
问题 I am wondering if anyone has tried using compression techniques for their LMDB files? Typically, lmdb files typically do not use any compression. I am wondering if anyone has successfully stored data in an lmdb using jpeg compression on lmdb and then used it for caffe. I need this because I am working on a developer board with very limited storage space. If so, can you please provide steps/code to do this? thanks 回答1: Caffe also supports HDF5 which supports compression. If your dataset is

Write jpeg file directly to lmdb [closed]

由 纵饮孤独 提交于 2019-12-24 06:07:39
问题 Closed . This question needs to be more focused. It is not currently accepting answers. Want to improve this question? Update the question so it focuses on one problem only by editing this post. Closed 2 years ago . I managed to write numpy arrays to lmdb, howewer solution is far from perfection, but actually my X is just jpg image, so my question is how to directly write jpeg file to lmdb? Seems like pycaffe doing similar thing but it use caffe specific Datum and I need some general solution
订阅 lmdb