用户您好,请详细描述您所遇到的问题:
系统软件版本:X3J3-PlatformSDK-PL2.1-V1.2.0-20221103
问题涉及的技术领域: camera 图像数据增强
问题描述:目前我们在x3平台上接入一款红外sensor,但从sensor输出的raw 数据的动态范围不够,所以想在raw数据域增加一个增强算法来提高动态范围,请问这个方案是否可行?如何实施?
用户您好,请详细描述您所遇到的问题:
系统软件版本:X3J3-PlatformSDK-PL2.1-V1.2.0-20221103
问题涉及的技术领域: camera 图像数据增强
问题描述:目前我们在x3平台上接入一款红外sensor,但从sensor输出的raw 数据的动态范围不够,所以想在raw数据域增加一个增强算法来提高动态范围,请问这个方案是否可行?如何实施?



您好,
如果是HAPI:
HB_VIN_GetDevFrame获取raw数据,经过算法处理,buff用完之后HB_VIN_ReleaseDevFrame释放,申请一个buff,处理之后的数据写到新申请的buff里,HB_VIN_SendPipeRaw给ISP。 可以参考demo,这几个接口的用法。
如果是VAPI:
hb_vio_get_data获取raw数据,经过算法处理,buff用完之后hb_vio_free_sifbuf释放,再用hb_vio_get_data获取回灌的buff,将算法处理后的数据写到该buff,然后调用hb_vio_run_raw回灌。可以参考demo,这几个接口的用法。

您好,
用hb_vio_get_data 接口获取 raw feeback buffer时,出现如下错误,这个是tuning_tool demo中的调用,我们添加的这个接口的调用也出现了同样的错误:
[DEBUG]["LOG"][vio_pipeline_grp.c:2389] [1126.222054]vio_pipeline_get_data[2389]: pipe(0) get_data type(8) In !!!
[DEBUG]["sif"][sif/sif.c:1469] [1126.222072]sif_entity_get_buf[1469]: Pipe(0) sif_entity_get_data(8) timeout(2000).
[ERROR]["sif"][sif/sif.c:1496] [1126.222090]sif_entity_get_buf[1496]: pipe(0)feedback buffer manager NULL
[INFO]["sif"][sif/sif.c:1515] [1126.222107]sif_entity_get_buf[1515]: Pipe(0) sif_entity_get_data(8) done.
[ERROR]["LOG"][vio_pipeline_grp.c:2431] [1126.222124]vio_pipeline_get_data[2431]: sif get fb src buf failed type(8) !!!
hb_vio_get_data feedback src buf failed.
请您帮忙分析一下,我们看不到vio的源码,多谢

您好,请问这里可以上传文件么,没找到上传的地方,以下是部分内容
log:
sensor init register: do nothing for iRay sensor
Hello, World! vio tool test start !
vio_cfg_file = ./vio_offline_raw12_512p.json
cam_cfg_file = ./hb_xj3dev_user.json
run_time = 10000
cam_index = 0
pipe_num = 1
data_type = 256
loop = 10000
need_m_thread = 0
need_get = 1
need_free = 1
need_gdc_fb = -1
hobotplayer_cfg_file = /etc/tuning_tool/dump_raw.json
Calculate or user config pipe_mask 0x1.
Test hb_vio_start_pipeline.
Test hb_vio_start_pipeline(0).
[15:26:05]dump_server_thread_start[274] viomonitor L: wait new socket connect.
pipe(0)data_type(256)raw feedback data in loop(10000)
Try to raw feedback src buf Pipe(0) data type(8) (sif_feed_src)
[ERROR]["sif"][sif/sif.c:1496] [521.372537]sif_entity_get_buf[1496]: pipe(0)feedback buffer manager NULL
[ERROR]["LOG"][vio_pipeline_grp.c:2431] [521.372561]vio_pipeline_get_data[2431]: sif get fb src buf failed type(8) !!!
hb_vio_get_data feedback src buf failed.
vio feedback failed. out here
pipe(0)data_type(256)raw feedback data in loop(9999)
Try to raw feedback src buf Pipe(0) data type(8) (sif_feed_src)
[ERROR]["sif"][sif/sif.c:1496] [521.372624]sif_entity_get_buf[1496]: pipe(0)feedback buffer manager NULL
[ERROR]["LOG"][vio_pipeline_grp.c:2431] [521.372635]vio_pipeline_get_data[2431]: sif get fb src buf failed type(8) !!!
hb_vio_get_data feedback src buf failed.
vio feedback failed. out here
pipe(0)data_type(256)raw feedback data in loop(9998)
Try to raw feedback src buf Pipe(0) data type(8) (sif_feed_src)
[ERROR]["sif"][sif/sif.c:1496] [521.372682]sif_entity_get_buf[1496]: pipe(0)feedback buffer manager NULL
[ERROR]["LOG"][vio_pipeline_grp.c:2431] [521.372693]vio_pipeline_get_data[2431]: sif get fb src buf failed type(8) !!!
hb_vio_get_data feedback src buf failed.
vio feedback failed. out here
pipe(0)data_type(256)raw feedback data in loop(9997)
Try to raw feedback src buf Pipe(0) data type(8) (sif_feed_src)
[ERROR]["sif"][sif/sif.c:1496] [521.372739]sif_entity_get_buf[1496]: pipe(0)feedback buffer manager NULL
[ERROR]["LOG"][vio_pipeline_grp.c:2431] [521.372750]vio_pipeline_get_data[2431]: sif get fb src buf failed type(8) !!!
vio json:
"pipeline0":{
"sif": {
"input": {
"dvp": {
"enable": 0,
"hsync_inv": 0,
"vsync_inv": 1,
"width": 3280,
"height": 2464,
"format": 0,
"pix_length": 2,
"enable_mux_out": 1,
"enable_frame_id": 1,
"enable_pattern": 0,
"set_init_frame_id": 100,
"set_mux_out_index": 0
},
"mipi": {
"enable": 1,
"ipi_channels": 1,
"mipi_rx_index": 1,
"width": 640,
"height": 512,
"format": 0,
"pix_length": 2,
"enable_mux_out": 1,
"enable_pattern": 0,
"enable_frame_id": 1,
"enable_bypass": 0,
"enable_line_shift": 0,
"enable_id_decoder": 0,
"set_init_frame_id": 0,
"set_line_shift_count": 0,
"set_bypass_channels": 1,
"set_mux_out_index": 0
},
"ddr_to_isp": {
"ddr_in_enable": 1,
"ddr_in_width": 640,
"ddr_in_height": 512,
"ddr_in_format": 0,
"ddr_in_pix_length": 2,
"ddr_in_buf_num": 6,
"ddr_raw_feedback_debug": 0
}
},
"output": {
"ddr": {
"ddr_output_enable" : 1,
"stride" : 960,
"ddr_output_buf_num": 8,
"mux_out_enable_index": 0,
"ddr_raw_dump_debug": 1
},
"isp": {
"isp_enable": 1,
"enable_flyby": 0,
"dol_exp_num": 1,
"enable_dgain ": 0,
"set_dgain_short": 0,
"set_dgain_medium": 0,
"set_dgain_long": 0
},
"ipu": {
"enable_flyby" : 0
},
"md": {
"enable": 0,
"path_sel" : 0,
"roi_top": 0,
"roi_left": 0,
"roi_width": 0,
"roi_height": 0,
"grid_step": 0,
"grid_tolerance": 0,
"threshold": 0,
"weight_decay": 0,
"precision": 0
}
}
},
"isp": {
"sensor_mode": 1,
"cfa_pattern": 0,
"bit_width": 12,
"temper_mode": 2,
"isp_raw_bypass": 0,
"test_pattern_enable" : 0,
"test_pattern_type": 0,
"out_width": 640,
"out_height": 512,
"output_dma_enable": 0,
"output_format": 0,
"output_buf_num": 6,
"isp_algo_state": 1,
"calib_mode": 1,
"calib_lname": "/etc/cam/libXcore_linear.so"
},
启动命令:
./tuning_tool -v ./vio_offline_raw12_512p.json -C ./hb_xj3dev_user.json -r 10000 -c 0 -p 1 -t 4 -l 10000 -m 0 -g 1 -f 1 -u -1 -H /etc/vio_tool/dump_raw.json

您好,我测试feedback 一张mipi raw12的图片,出现以下错误,请帮忙指导一下,谢谢:
pipe(0)data_type(256)raw feedback data in loop(10000)
Try to raw feedback src buf Pipe(0) data type(8) (sif_feed_src)
normal pipe_id (0)type(8)frame_id(0)buf_index(0)w x h(640x512)
dol_num is 1
feedback file name./r.raw
raw img_addr.addr[0] = 0x7faed17000, size 491520
read raw file size 491520 [ERROR]["LOG"][vio_pipeline_grp.c:2632] [1725.708954]vio_pipeline_raw_feedback[2632]: pipe(0)feedback is not configed.or fd & dump op
raw feed back failed.
vio feedback failed. out here

您好,您现在测试程序中用的回灌接口是hb_vio_raw_feedback,这个接口比较旧,建议用hb_vio_run_raw接口回灌raw,流程:
然后使用接口hb_vio_run_raw回灌Raw. 这样的话ddr_raw_dump_debug和ddr_raw_feedback_debug可以同时为1.

您好,我现在测试从sif 获取数据,然后什么都不做,直接再feedback,遇到如下问题:
1 从hobotplayer上只能收到6帧的yuv数据,之后就不再吐yuv了
2 tuning tool 跑起来后有如下错误输出:
先是以下错误:
hobotplayer_cfg_file = /etc/tuning_tool/dump_raw.json
Calculate or user config pipe_mask 0x1.
Test hb_vio_start_pipeline.
Test hb_vio_start_pipeline(0).
[DIAG_WARN]diag_lib_app.cpp:131] Please start diag service first
[15:26:05]dump_server_thread_start[274] viomonitor L: wait new socket connect.
image data hance enter
image data hance leave!!!
pipe(0)Get DS2 data in loop operation. loop(10000)
Try to get Pipe(0) data type(2) (ipu_ds2),status(0),time(0)
[ERROR]["vio_devop"][utils/dev_ioctl.c:280] [3476.140488]dev_get_buf_timeout[280]: pipe(0)TIME OUT ,sec(3476)usec(140487)!
[ERROR]["vio_devop"][utils/dev_ioctl.c:283] [3476.140615]dev_get_buf_timeout[283]: pipe(0)Get buf sem_timedwait failed Connection timed out!
[ERROR]["vio_bufmgr"][utils/hb_vio_buffer_mgr.c:2011] [3476.140725]buf_mgr_print_qcount[2011]: Mgr(2)state:Total(6)Avail(3)Process(3)Done(0)Repro(0)User(0).
[ERROR]["LOG"][vio_pipeline_grp.c:2411] [3476.140815]vio_pipeline_get_data[2411]: ipu get buf failed type(2) !!!
pipe(0)get data type(2) failed!
image data hance enter
[ERROR]["vio_devop"][utils/dev_ioctl.c:280] [3476.141955]dev_get_buf_timeout[280]: pipe(0)TIME OUT ,sec(3476)usec(141955)!
[ERROR]["vio_devop"][utils/dev_ioctl.c:283] [3476.142046]dev_get_buf_timeout[283]: pipe(0)Get buf sem_timedwait failed Connection timed out!
[ERROR]["vio_devop"][utils/dev_ioctl.c:280] [3476.142399]dev_get_buf_timeout[280]: pipe(0)TIME OUT ,sec(3476)usec(142399)!
[ERROR]["vio_devop"][utils/dev_ioctl.c:283] [3476.142475]dev_get_buf_timeout[283]: pipe(0)Get buf sem_timedwait failed Connection timed out!
[ERROR]["vio_bufmgr"][utils/hb_vio_buffer_mgr.c:2011] [3476.142548]buf_mgr_print_qcount[2011]: Mgr(2)state:Total(6)Avail(3)Process(3)Done(0)Repro(0)User(0).
[ERROR]["LOG"][vio_pipeline_grp.c:2411] [3476.142627]vio_pipeline_get_data[2411]: ipu get buf failed type(2) !!!
[15:26:05]get_yuv_data[286] viomonitor E: -------get yuv failed---------
image data hance leave!!!
输出6帧以后,在以上错误之外又出现以下错误:
[ERROR]["sif"][sif/sif.c:1490] [3491.379636]sif_entity_get_buf[1490]: pipe(0)can not get buffer from available queue
[ERROR]["LOG"][vio_pipeline_grp.c:2431] [3491.379746]vio_pipeline_get_data[2431]: sif get fb src buf failed type(8) !!!
我加下raw dump 和 feedback 的代码如下:
static int raw_data_hance_process(work_info_t * info)
{
int ret = 0, get_ctx_rt = 0;
hb_vio_buffer_t *sif_raw = NULL;
struct timeval time_now = { 0 };
struct timeval time_next = { 0 };
int size = -1;
int time_ms = 0;
unsigned char *buffer_raw = NULL;
unsigned char *feedback_raw = NULL;
hb_vio_buffer_t sif_raw_feedback = {0};
printf("image data hance enter\n");
ret = hb_vio_get_data(info->pipe_id, HB_VIO_SIF_RAW_DATA,
sif_raw);
if (ret < 0){
printf("get raw data failed, return!");
free(sif_raw);
sif_raw = NULL;
return -1;
}
size = sif_raw->img_addr.stride_size * sif_raw->img_addr.height;
ret = hb_vio_get_data(info->pipe_id, HB_VIO_SIF_FEEDBACK_SRC_DATA,
&sif_raw_feedback);
if(ret < 0) {
printf("hb_vio_get_data feedback src buf failed.\n");
free(sif_raw);
sif_raw = NULL;
return -1;
}
memcpy(sif_raw_feedback.img_addr.addr[0], sif_raw->img_addr.addr[0], size);
HB_SYS_CacheFlush(sif_raw_feedback.img_addr.paddr[0],
sif_raw_feedback.img_addr.addr[0],
sif_raw_feedback.img_info.size[0]);
hb_vio_free_sifbuf(info->pipe_id, sif_raw);
ret = hb_vio_run_raw(info->pipe_id, &sif_raw_feedback, 100);
if (ret < 0){
printf("raw data feedback failed.\n");
free(sif_raw);
sif_raw = NULL;
return -1;
}
if (sif_raw) {
free(sif_raw);
sif_raw = NULL;
}
printf("image data hance leave!!!\n""");
return 0;
}
请帮忙指导一下,谢谢

首先sif_raw是通过get_data获取的,要通过hb_vio_free_sifbuf释放buff,不是free。
下面是获取raw:
hb_vio_buffer_t sif_raw = {0};
ret = hb_vio_get_data(pipe_id, info->data_type, &sif_raw);
if (ret < 0) {
printf("hb_vio_get_data sif_raw error!!!\n");
}
回灌流程:
还有就是回灌的buff是通过
hb_vio_get_data(info->pipe_id, HB_VIO_SIF_FEEDBACK_SRC_DATA, &src_raw_buff)获取的,
将上面获取的sif_raw 结构体地址的数据copy到src_raw_buff,然后调用hb_vio_free_sifbuf(pipe_id, &sif_raw)释放。
通过hb_vio_run_raw(info->pipe_id, &sif_raw_src, 2000)回灌。 时间设置为2s.
回灌之后回灌的buff也要通过hb_vio_free_sifbuf(info->pipe_id, &sif_raw_src)释放。

谢谢您的细心指导,现在可以连接出图了,但是yuv的出图很慢,sif raw dump + feedback 的耗时大概在30~40ms, 请教一下为什么yuv的出图会这么慢,大概要2s左右hobotplayer才能更新一张,如下图所示:
以下是log 输出,还想请教一下,这里的ipu get buffer错误是由什么引起的,这里还有改进空间么?
image data enhance enter
[ERROR]["vio_devop"][utils/dev_ioctl.c:280] [11834.728105]dev_get_buf_timeout[280]: pipe(0)TIME OUT ,sec(11834)usec(728105)!
[ERROR]["vio_devop"][utils/dev_ioctl.c:283] [11834.728199]dev_get_buf_timeout[283]: pipe(0)Get buf sem_timedwait failed Connection timed out!
[ERROR]["vio_devop"][utils/dev_ioctl.c:280] [11834.734538]dev_get_buf_timeout[280]: pipe(0)TIME OUT ,sec(11834)usec(734538)!
[ERROR]["vio_devop"][utils/dev_ioctl.c:283] [11834.734662]dev_get_buf_timeout[283]: pipe(0)Get buf sem_timedwait failed Connection timed out!
[ERROR]["vio_bufmgr"][utils/hb_vio_buffer_mgr.c:2011] [11834.734753]buf_mgr_print_qcount[2011]: Mgr(2)state:Total(6)Avail(3)Process(3)Done(0)Repro(0)User(0).
[ERROR]["LOG"][vio_pipeline_grp.c:2411] [11834.734837]vio_pipeline_get_data[2411]: ipu get buf failed type(2) !!!
[15:26:05]get_yuv_data[286] viomonitor E: -------get yuv failed---------
image data enhance time cost 39 ms
image data enhance leave!!!
pipe(0)Get DS2 data in loop operation. loop(9953)
Try to get Pipe(0) data type(2) (ipu_ds2),status(0),time(0)
[ERROR]["vio_devop"][utils/dev_ioctl.c:280] [11837.766953]dev_get_buf_timeout[280]: pipe(0)TIME OUT ,sec(11837)usec(766953)!
[ERROR]["vio_devop"][utils/dev_ioctl.c:283] [11837.767081]dev_get_buf_timeout[283]: pipe(0)Get buf sem_timedwait failed Connection timed out!
[ERROR]["vio_bufmgr"][utils/hb_vio_buffer_mgr.c:2011] [11837.767172]buf_mgr_print_qcount[2011]: Mgr(2)state:Total(6)Avail(3)Process(3)Done(0)Repro(0)User(0).
[ERROR]["LOG"][vio_pipeline_grp.c:2411] [11837.767264]vio_pipeline_get_data[2411]: ipu get buf failed type(2) !!!
pipe(0)get data type(2) failed!

IPU报错原因:
1. 有可能是vio json中使能的IPU chn和命令行中对应的chn不一致导致的。
2. IPU 没有收到数据报错。
下面几个命令可以看每个模块的帧率。
cat /sys/devices/platform/soc/a4001000.sif/fps
cat /sys/devices/platform/soc/b3000000.isp/fps
现在获取raw,经过你算法处理,再去copy,回灌,这条路比较长,比较耗时。
你的网络带宽是多少?百兆还是千兆?是不是网络传的就很慢。

我们之前有个同事用过同样方式,可以跑的,测试程序我发给您,这个不能贴文件,那我就发给你一部分,下面部分是在一个线程里执行的,camera start之后起个线程去做这个事情。您对比一下。
int raw_run_feedback_func(work_info_t * info, int loop_count)
{
#define RAW_FEED_USE_SIF 1
int ret = 0;
hb_vio_buffer_t sif_raw_src = {0};
hb_vio_buffer_t sif_raw_src_in = {0};
char file_name[100] = { 0 };
static int raw_idx = 0;
static int file_parsed = 0;
FILE *inputFp = NULL;
if (!file_parsed) {
file_parsed = 1;
ret = file_name_collect(raw_dir, raw_img_path, &raw_img_num);
printf("Test feedback data raw file(%d) \n", raw_img_num);
}
if (info->data_type != HB_VIO_SIF_FEEDBACK_SRC_DATA) {
printf("Test Not sif feedback data type (%d)\n", info->data_type);
return -1;
}
if (need_get) {
printf("Test Try to raw_run_feedback src buf Pipe(%u) data type(%d) (%s)\n",
info->pipe_id, info->data_type, buf_name[info->data_type]);
#if (RAW_FEED_USE_SIF == 0)
// 0 get a empty raw buf from vio
ret = hb_vio_get_data(info->pipe_id, HB_VIO_SIF_FEEDBACK_SRC_DATA,
&sif_raw_src);
if(ret < 0) {
printf("hb_vio_get_data feedback src buf failed.\n");
goto err;
}
normal_buf_info_print(&sif_raw_src);
if (raw_idx >= raw_img_num) {
raw_idx = 0;
printf("Test raw was feedback done, roll-back to zero.\n");
}
strcpy(file_name, raw_img_path[raw_idx]);
printf("Test feedback raw file(%d) name %s\n", raw_idx, file_name);
inputFp = fopen(file_name, "rb");
if (inputFp == NULL) {
printf("Test raw file(%s) open failed\n", file_name);
return -1;
}
printf("Test feed raw img_addr.addr[0] = %p, size %d \n",
sif_raw_src.img_addr.addr[0],
sif_raw_src.img_info.size[0]);
int count = fread(sif_raw_src.img_addr.addr[0], 1,
sif_raw_src.img_info.size[0], inputFp);
printf("Test read raw file size %d \n", count);
fclose(inputFp);
#elif (RAW_FEED_USE_SIF == 1)
ret = hb_vio_get_data(info->pipe_id, HB_VIO_SIF_FEEDBACK_SRC_DATA,
&sif_raw_src);
if(ret < 0) {
printf("hb_vio_get_data feedback src buf failed.\n");
goto err;
}
normal_buf_info_print(&sif_raw_src);
if (raw_idx >= raw_img_num) {
raw_idx = 0;
printf("Test raw was feedback done, roll-back to zero.\n");
}
hb_vio_get_data(info->pipe_id, HB_VIO_SIF_RAW_DATA, &sif_raw_src_in);
if(ret < 0) {
printf("hb_vio_get_data raw src buf failed.\n");
goto err;
}
normal_buf_info_print(&sif_raw_src_in);
memcpy(&sif_raw_src.img_addr, &sif_raw_src_in.img_addr, sizeof(sif_raw_src.img_addr));
if (raw_idx >= raw_img_num) {
raw_idx = 0;
printf("Test raw was feedback done, roll-back to zero.\n");
};
#endif
raw_idx++;
// 3 feedback to isp ddr in
printf("Test call hb_vio_run_raw, pipe %d\n", info->pipe_id);
ret = hb_vio_run_raw(info->pipe_id, &sif_raw_src, -1);
// 4 dump isp output here
if(ret < 0) {
printf("raw feed back failed.\n");
goto err;
} else {
#if 0
if(need_dump) {
ret = hb_vio_get_data(info->pipe_id, HB_VIO_ISP_YUV_DATA, &isp_yuv_dst);
if (ret < 0) {
printf("hb_vio_get_data isp_yuv data error!!!\n");
} else {
if (access(ISP_DUMP_PATH, 0)) {
ret = mkdir(ISP_DUMP_PATH, 0777);
if (ret < 0) {
printf("mkdir %s for isp dump failed !\n", ISP_DUMP_PATH);
}
}
normal_buf_info_print(&isp_yuv_dst);
size = isp_yuv_dst.img_addr.stride_size * isp_yuv_dst.img_addr.height;
printf("yuv stride_size(%u) w x h%u x %u, size %d\n",
isp_yuv_dst.img_addr.stride_size,
isp_yuv_dst.img_addr.width, isp_yuv_dst.img_addr.height, size);
sprintf(file_name, "%s/pipe%d_%ux%u_frame_%03d.yuv",
ISP_DUMP_PATH,
info->pipe_id,
isp_yuv_dst.img_addr.width,
isp_yuv_dst.img_addr.height,
isp_yuv_dst.img_info.frame_id);
dumpToFile2plane(file_name, isp_yuv_dst.img_addr.addr[0],
isp_yuv_dst.img_addr.addr[1], size, size/2);
hb_vio_free_ispbuf(info->pipe_id, &isp_yuv_dst);
printf(" Pipe(%u) hb_vio_free_ispbuf dump get done type(%d) (%s) \n",
info->pipe_id, info->data_type, buf_name[info->data_type]);
}
}
#endif
}
printf(" Pipe(%u) raw feedback(%d) done type(%d) (%s) \n",
info->pipe_id, raw_idx, info->data_type, buf_name[info->data_type]);
if (need_free) {
usleep(30000);
hb_vio_free_sifbuf(info->pipe_id, &sif_raw_src);
#if (RAW_FEED_USE_SIF == 1)
sprintf(file_name, "%s/pipe%d_plane%d_%ux%u_frame_%03d.raw",
RAW_DUMP_PATH,
dump_info.ctx_id,
i,
dump_info.raw.xres[i],
dump_info.raw.yres[i],
dump_info.raw.frame_id);
dumpToFile(file_name, dump_info.raw.addr[i], dump_info.raw.size[i]);
hb_vio_free_sifbuf(info->pipe_id, &sif_raw_src_in);
#endif
}
raw_idx++;
}
return 0;
err:
printf("vio feedback failed. out here\n");
return -1;
}
